We have a set of three standalone Nginx-instances running on Ubuntu, these are then load balanced in a Fortigate firewall using a virtual server configuration where load distribution is done using round robin load balancing. This means when going to a single IP, you might be redirected to any of these Nginx-instances. We have a nightly job that will copy all modified files from the primary Nginx ("01") to the other two, including Nginx-configurations and SSL certificates. The issue is that when doing the domain validation, certbot sometimes fails to verify the domain since the Nginx instance it "ends up on", is not the same that we try to request the certificate from. Any tips on how this can be done while still retaining autorenewal for certificates?
My domain is:
handverksdata-integrator.no
I ran this command:
sudo certbot certonly --nginx
Running this command results in this output, since it is redirected to another Nginx-instance than the "master" that I'm attempting to run this from:
Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: androidtest.handverksdata-integrator.no
Type: unauthorized
Detail: 109.228.156.36: Invalid response from http://androidtest.handverksdata-integrator.no/.well-known/acme-challenge/hz3ZUr1sqjy3S9onqYOq45nKVmfpejptOpUICi7ryPg: 404
The --nginx authenticator uses an HTTP Challenge. Certbot only prepares the nginx system on the server it runs as to handle that. The Challenge actually comes from the Let's Encrypt Server (Certbot is the ACME Client).
And, yes, a round-robin balancer won't necessarily send the inbound HTTP challenge to that same system.
There are various ways to handle this. Do you have a way to direct an HTTP request to server1 where Certbot is running? Like with a unique DNS hostname? Example: server1.handverksdata-integrator.no
If so, the nginx conf for your 3 servers could redirect the original incoming challenge to that name. Let's Encrypt follows redirects. See: Challenge Types - Let's Encrypt
You make a new server block in nginx for that domain name just to handle this redirected request. You also need to switch to using --webroot rather than --nginx as Certbot's --nginx only updates the server block for the original domain name.
Or, use a DNS Challenge. That is also described at that link. For automation your DNS provider needs to support an API for adding/deleting the necessary TXT record. Your DNS provider also needs to be supported by your ACME Client (Certbot in this case). Maybe another ACME Client like lego (link here) would support it?
Please review the notes about DNS Challenge API credentials in that LE Challenge Types page
There are other ways but they are likely variations on this theme. For example you can setup a Certbot --standalone to handle a redirected request. Or switch DNS providers to one that Certbot supports (if it does not already).
Yet another option is to use certbot's --pre-hook and --post-hook to copy the challenge file to the other servers, so that no matter which server the request gets routed to it can respond correctly.
Thanks for the reply! I am checking with our DNS provider if it's possible to do via an API. Unfortunately we cannot redirect the incoming requests to a specific Nginx-instance.
This sounds interesting, do you have practical experience with using that? I will have to read up on the documentation to see how it could be implemented.
No, I don't have any experience with it myself. The pre-hook and post-hook can be used to run a shell script that you write, with environment variables set for the domains being updated and such.
I assume that it's at a part in the process that would allow for you to copy the file over, but I might be making some wrong assumptions.
Another option is to use "manual" instead of "nginx", where you get control over the entire process and run your own authentication script for whatever you need to do to satisfy the challenge.
If your DNS provider does not have a supported API you could add a static CNAME for the _acme-challenge record in that DNS to a different DNS system that does have an API
Thanks, yes it seems like it might be the way to go. Would certbot still be able to automatically renew certificates this way? Unfortunately our DNS provided doesn't have an API that supports automatically adding TXT-records.
Note the server you currently use --nginx automatically reloads nginx after getting a new cert. The --manual method will not. A --deploy-hook can reload nginx after getting a fresh cert. See the Certbot docs for that option.
Another option to explore is to see if you can have your load balancer always route traffic for /.well-known/acme-challenge/ on port 80 to your primary server. (Or maybe, failing that, always route port 80 to only your primary server, even though 443 goes to all your servers. If the only thing being served on 80 is a redirect to https and the occasional certificate challenge, then maybe it doesn't have as much load that needs balancing.)
Wonderful, this ended up being the easiest solution actually. Adding the location of /.well-known/acme-challenge/ with a proxy_pass to the first Nginx-instance on port 80, it now works perfectly. Thank you!