Nginx is running on all three server delivering the same content. Best way would be that the certs are generated by one server e.g. 1.2.3.4, but a request is only served by one-third of the cases from server 1.
Hence certbot certonly --nginx throws the following error in two-third of the cases:
Failed authorization procedure. www.example.com (tls-sni-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect validation certificate for tls-sni-01 challenge. Requested acme.invalid from 1.2.3.5:443.
In my opinion, the use of the webroot plugin (and thus the http-01 challenge) in combination with a redirect for /.well-known/acme-challenge/ to a separate hostname (e.g., acme-validator.example.com) which has just one IP address would be the answer to your problem.
Of course, the client should run on the server with the IP of acme-validator.example.com and you’d need a way to distribute the certificate/private key to the other servers.
If you only run certbot on one machine and copy the certificates to the rest, you could also set up a proxy rule on the other two nginx servers so even if the validation authority contacts one of the other two servers it will forward the request to the right one.
You could also set up a proxy or redirect loop so each server could use HTTP-01 validation and maintain independent certificates. If the file exists, it’s served; if it doesn’t, server 1 redirects to server 2, server 2 redirects to server 3, server 3 redirects to server 1.
Or you could use DNS-01 validation. It’s perfectly suited for situations like this, but may take more work to set up.
Thanks for your help guys. Running certbot on one machine with redirects to this host and http-01 challenge worked. What is the best way to distribute the certs among the rest of the servers?
Each server uses the same private key, so the same cert can be re-used everywhere.
Have the one server that runs the renewal publish the certificate to an accessible URL or file share (as part of renewal hook).
The remainder of the servers do a nightly fetch of the certificate + graceful web server reload via crontab.
It has been reliable for me and I don't think that it's a security problem because no sensitive key material is ever moved across the network.
The one enhancement I would make is that the servers which receive the new certificate should do a validity check (name + validity period + issuer + trust) using openssl, in case some smartass decides to upload a bogus cert. I haven't got around to it.