I followed this topic which helped me weed out my problem and hoping this saves someone else pain with this.
After creating a certificate for the first time for a mail2 node bearing the full DNS node name of mail2.example.com (changed real domain for privacy of real domain) I found it would not renew the certificate despite working the first time. I followed topic 218002 for about 4 days of attacking my issue but nothing worked.
Finally I realised, thanks to 218002, that the extra entries added by certbot to my conf file were the problem. The renew process was hitting the mail2 node behind the reverse-proxy and not using the local /usr/share/nginx/html directory on the reverse-proxy server.
In this example you will see I have commented out the lines added by the original certbot process and now the node.domain_name works:
That may have helped your situation but that is not a normal requirement when using the --nginx option for Certbot.
That option dynamically inserts code into your nginx to handle the challenge. The code is removed afterwards.
What you've done is disable your port 80 server block. Certbot's nginx then creates a default server block (if no other is available) to satisfy the challenge.
I'd call this a work-around of something gone wrong in your overall config rather than a proper solution. My first guess is you may have used different Certbot methods at different times which gave you a mis-match between Certbot's renewal config file and your nginx. For example, by mixing --nginx sometimes and certonly --webroot at others. Other things could cause a similar initial problem.
Yes, fair. But, from the "managed by Certbot" comments in the server block they probably used the --nginx plugin option as the authenticator. They for sure used it at least as --installer.
The Certbot --nginx authenticator adds config settings directly into the matching server block(s) to allow the ACME challenge, reloads nginx, requests the cert, and then removes those temp settings and reloads nginx again.
The /.well-known/acme-challenge/ path is only viable for that brief time. It is possible to debug that sequence but it takes some care. The --debug-challenges -v is one option to help with that.
I think considerably more went wrong with their system and I hope people don't follow their work-around to address routine problems. I expect they will suffer future problems from that work-around.
