I'm writing this because of an issue that I faced, and there were no good answers to it on any of the forums, however I managed to solve it and wanted to share it with you.
My setup is as such:
OS : Debian 9 , Kernel 4.9.0-14-amd64 on GCP
Web server: nginx/1.18.0
What this server is doing : Hosting many virtual hosts, some of them are static sites, others are php based sites and some are Django sites.
Problem : 3 months ago, the certbot autorenew stopped working. I was also not able to obtain new certificates for new sites. The error was the usual 'nginx 404' that a lot of people were getting.
Failed authorization procedure. example.domain (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://example.domain/.well-known/acme-challenge/BgxplVnHSmEVj3WLwnAsr3CNaTG6s_RHEUKXwdvSTOE [xxx.xxx.xxx.xxx]: 404
I tried just about everything under the sun and couldn't get it to work. Finally to debug the issue, I ran the command
# certbot certonly --test-cert --dry-run -d example.domain and at the same time ran a
tail -f on the nginx site, the access log as well as the error log to check out what was going on. The site file was being modified correctly, nginx was being reloaded correctly, however the access logs were still throwing 404's however when I visited the page with my browser a second later, it loaded fine. That meant nginx was probably taking a tad bit longer to restart. I have some heavy modsecurity custom rulesets running on the server which was slowing down the reload of nginx.
I figured if I could somehow delay the request from Let's Encrypt's servers, then it would work. So I cd'ed into the
/usr/lib/python3/dist-packages/certbot_nginx directory and ran
grep -ri sleep. I found the culprit in a file called
configurator.py. In the function
def nginx_restart(nginx_ctl, nginx_conf):, there's a
time.sleep(1) - it's towards the end of the file. I just changed that to
time.sleep(4) and everything worked again.
This had me pulling my hair out for months and I thought sharing the solution might help someone out there. Hope it helps!!