I need something very basic explained to me. Please forgive my ignorance. My server is httpd on linux.
To point the web server to the temporary location that contains the challenge file certbot modifies the httpd.conf file.
writing a pre config file with text:
RewriteEngine on
RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted
</Directory>
<Location /.well-known/acme-challenge>
Require all granted
</Location>
AFAIU this modification can only be enabled after the httpd.conf file is re-read by httpd, which can be done in three ways: stop/start, restart, or reload. For whatever reason certbot fails to perform this action. There are no mentions of any failures even with verbosity on -vvv, and I cannot find the mechanism that performs the restart/reload. All I get is:
Certbot failed to authenticate some domains
Invalid response : 404
which is not surprising, since the httpd does not know anything about the configuration change without restart/reload.
I can confirm that the challenge file is created successfully, and the httpd.conf changes are also made successfully, because if I add debug-challenge option to the command line, I can wait for the prompt, then open another terminal window and manually perform the httpd restart, then press Enter in the certbot window, and the process completes successfully. But this defies the certbot's usefulness as an automation tool.
@transmodus, Certbot tries to detect your OS and then tries to issue an appropriate command to restart your Apache server. There are some builtin ways that it will do this, but it could get it wrong, especially if your Apache is in a nonstandard location for your OS or was installed in an unusual way. I agree with the other commenters that seeing the command you ran and the log file would be helpful.
The detected OSes are listed in
If you don't see yours there, or you know that your httpd isn't where it typically is on that OS, then you may have to do something extra here.
I just took a look really quickly and it (surprisingly) looks like it might not be logged if Certbot thinks it succeeded, only if it thinks it failed. I would think it would be helpful to have it log the reload command it tried to use.
Yeah from what I see it seems that certbot is convinced that everything worked...
So there IS a reload command somewhere that is supposed to reload httpd? Where is it? Can it be changed to "restart" instead?
As mentioned above the EXACT same configuration works fine when I stop and reload the conf files before proceeding. Doesn't seem to me that the http/https has anything to do with that. And the response is 404, file not found, so the connection is made successfully over https.
Yes, I'm game. Tell me what and where to add.
BTW, maybe this is the problem?
[root@cmp-l2p-web-qa-04 wtl.d]# which apache2ctl
/usr/bin/which: no apache2ctl in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/var/lib/snapd/snap/bin:/root/bin)
[root@cmp-l2p-web-qa-04 wtl.d]# which apachectl
/sbin/apachectl
I'll make a symlink for apache2ctl and try the renewal again.
I'm not sure that would fix it, as it would log an error if the command couldn't be found.
I'm also interested to see what the Apache log would show. Usually I'd use tail -f to open the log and show the lines being added in realtime and then run e.g. Certbot to see what Apache does.
Here is where I think information about reloading apache should appear:
With ' --debug-challenges' :
2022-06-07 10:24:16,542:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd/conf.d/mod_cluster.conf
2022-06-07 10:24:16,543:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com-le-ssl.conf
2022-06-07 10:24:16,543:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/conf.d/mod_cluster.conf
2022-06-07 10:24:16,544:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com.conf
2022-06-07 10:24:19,828:DEBUG:certbot._internal.display.obj:Notifying user: Challenges loaded. Press continue to submit to CA.
The following URLs should be accessible from the internet and return the value
mentioned:
URL:
http://r3test.paycosmos.com/.well-known/acme-challenge/suUWNtCPWRb3BfbAwX78v8FE6jwYr2HkmSMV6HPfjGw
Expected value:
suUWNtCPWRb3BfbAwX78v8FE6jwYr2HkmSMV6HPfjGw.tnPLcddM602NajLZJBpYWVPVn3E93ubwf0htgR_QWRQ
<I PRESS ENTER, POSSIBLY RESTARTING HTTPD IN A SEPARATE TERMINAL>:
2022-06-07 10:24:20,700:DEBUG:acme.client:JWS payload:
b'{}'
The same operation without ' --debug-challenges' :
2022-06-07 10:12:24,189:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com-le-ssl.conf
2022-06-07 10:12:24,190:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/conf.d/mod_cluster.conf
2022-06-07 10:12:24,190:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd-wtl/wtl.d/r3test.paycosmos.com.conf
2022-06-07 10:12:24,190:DEBUG:certbot.reverter:Creating backup of /opt/rh/jbcs-httpd24/root/etc/httpd/conf.d/mod_cluster.conf
<THE HTTPD SHOULD GET RELOADED/RESTARTED AT THIS POINT>
2022-06-07 10:12:27,489:DEBUG:acme.client:JWS payload:
b'{}'
2022-06-07 10:12:27,492:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/117136456736/VecRZw:
Yes, probably. But I'm more interested in the Apache logs around that time. Not sure how to continue debugging this further without modifying the code to the certbot-apache plugin, but that would have to be done on your computer.
It's kinda hard to tell someone through a forum to do this, do that. Something that one would test out in a few minutes, could take hours unfortunately.
Best way is probably to know how you installed your Apache including its configuration and to try to reproduce it locally.
Also, looking at your own Apache logs doesn't require anything but you looking.