I have been using certbot with webroot for years now and it worked very well. The actual web server is nginx, but I don't use the nginx plugin as the certs are also used for other purposes like mail.
To make matters more complicated, the system I am running certbot and nginx on has port 80 occupied, so I've set up NAT in the router that maps www.rna.nl:80 to an internal nginx on port 8081.
But recently I have started to have trouble with this approach and it seems the combination of webroot and nginx. I suspect it may have to do with nginx becoming too slow to refresh itself and thus erroneously reporting 404 on a file that is really there. I.e. I see 404 failures in nginx access log, but at the same time I see certbot has correctly created those files. And while it works for one it doesn't work for another server that sits in the same nginx.
So, I was thinking, can I move to standalone? Tell standalone to use port 8081 and more importantly, migrate the existing webroot config for my certificates to standalone?
If you are using --webroot as your authenticator plugin and --nginx as the installer plugin, the reload speed of nginx wouldn't matter. You would never see challenge validation errors, unless there was a mistake in the nginx configuration.
If you can reproduce the issue with a command like this, we could help you work through it:
# certbot -v --dry-run --cert-name www.rna.nl -n renew
Saving debug log to /opt/local/var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /opt/local/etc/letsencrypt/renewal/www.rna.nl.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Simulating renewal of an existing certificate for www.rna.nl and mail.rna.nl
Performing the following challenges:
http-01 challenge for mail.rna.nl
http-01 challenge for www.rna.nl
Using the webroot path /opt/local/share/nginx/www.rna.nl for all unmatched domains.
Waiting for verification...
Challenge failed for domain mail.rna.nl
Challenge failed for domain www.rna.nl
http-01 challenge for mail.rna.nl
http-01 challenge for www.rna.nl
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: mail.rna.nl
Type: unauthorized
Detail: 213.125.118.53: Invalid response from http://mail.rna.nl/.well-known/acme-challenge/UPS4CWV0p3RrN4y4QfzL3iSSrzIyOjQ69A32goC_5gE: 404
Domain: www.rna.nl
Type: unauthorized
Detail: 213.125.118.52: Invalid response from http://www.rna.nl/.well-known/acme-challenge/Wyc2SAHKSgSMFguMUTkQpkEz25SNm_jR7ecPfzyuf1E: 404
Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
Cleaning up challenges
Failed to renew certificate www.rna.nl with error: Some challenges have failed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
/opt/local/etc/letsencrypt/live/www.rna.nl/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Before and while this is running, the actual webroot location gets the files stored, so nginx should be able to find them:
# for i in www.rna.nl/.well-known/acme-challenge/*; do (ls -l "$i"; echo -n "$i: "; cat "$i"; echo ""); done
-rw-r--r-- 1 root staff 0 Dec 6 21:15 www.rna.nl/.well-known/acme-challenge/letsdebug-test
www.rna.nl/.well-known/acme-challenge/letsdebug-test:
# for i in www.rna.nl/.well-known/acme-challenge/*; do (ls -l "$i"; echo -n "$i: "; cat "$i"; echo ""); done
-rw-r--r-- 1 root staff 87 Dec 7 12:00 www.rna.nl/.well-known/acme-challenge/UPS4CWV0p3RrN4y4QfzL3iSSrzIyOjQ69A32goC_5gE
www.rna.nl/.well-known/acme-challenge/UPS4CWV0p3RrN4y4QfzL3iSSrzIyOjQ69A32goC_5gE: UPS4CWV0p3RrN4y4QfzL3iSSrzIyOjQ69A32goC_5gE.rh2TKRH6lIyDYCyR8UsM2ABhLdghiTnWvTr176AMCuI
-rw-r--r-- 1 root staff 87 Dec 7 12:00 www.rna.nl/.well-known/acme-challenge/Wyc2SAHKSgSMFguMUTkQpkEz25SNm_jR7ecPfzyuf1E
www.rna.nl/.well-known/acme-challenge/Wyc2SAHKSgSMFguMUTkQpkEz25SNm_jR7ecPfzyuf1E: Wyc2SAHKSgSMFguMUTkQpkEz25SNm_jR7ecPfzyuf1E.rh2TKRH6lIyDYCyR8UsM2ABhLdghiTnWvTr176AMCuI
-rw-r--r-- 1 root staff 0 Dec 6 21:15 www.rna.nl/.well-known/acme-challenge/letsdebug-test
www.rna.nl/.well-known/acme-challenge/letsdebug-test:
Do you want me to add the letsencrypt debug log? Actually, what happens is that nginx cannot find the file that certbot has definitely written and more permanent files with the same permissions are easily delivered by nginx. So, it does seem some sort of nginx problem or certbot-nginx race condition.
Difference between both sites is that albus.rna.nl is empty while www.rna.nl has a bit of web site so might load slower on some sort of renewal by nginx.
I hope it is not something really silly on my part...
This will pause Certbot right before it submits the challenges to the certificate authority.
While it is paused, you can inspect the contents of the directory and try to manually request the URLs yourself. Check the nginx access and error logs to see if anything interesting comes up.
The contents of the nginx server block for this virtual host would help, as well as the contents of /opt/local/etc/letsencrypt/renewal/www.rna.nl.conf.
I was busy getting all the info together, and testing more and then I noticed that I am an utter moron. It was the NAT port forward rule after all...
Basically, I first had www.rna.nl and mail.rna.nl right (WAN IPs port forwarded to internal port 8081) then I had albus.rna.nl right (different WAN IP port forward to port 8082) but I had only one rule. So when I fixed it for one ("oh wait, the port is wrong, it should be 8081", "oh wait, the port is wrong, it should be 8082"), I broke it for the other.
Why was this so hard to spot? Because at the same time I was having trouble with the packet filter of the ever more unstable macOS Monterey that runs nginx. This also played a role when I tested on the LAN. And because both had the same letsdebug-test file I never noticed that www.rna.nl got served the one from albus.rna.nl or vice versa.
Problem in chair, not in computer. Thank you very much for trying to help, but it turned out the problem was me. Actually having the pause and seeing that the file was here for a long while but still not found gave me the answer.
The order matters only if you're providing different webroots fot different domains:
--webroot-path WEBROOT_PATH, -w WEBROOT_PATH
public_html / webroot path. This can be specified multiple times to handle different domains; each domain will have
the webroot path that preceded it. For instance: -w /var/www/example -d example.com -d www.example.com -w /var/www/thing -d thing.net -d m.thing.net