I searched before posting, and I found many people with similar questions, but I thought my scenario was different enough to warrant a new post.
I run a service that hosts about 800 websites. We use SSL termination with HAProxy, and we want to use the fault tolerance of Route 53 for DNS management of multiple HAProxy servers.
Current set up (and hoping to have multiple proxies):
Amazon Route 53 DNS
Multiple HAProxy servers SSL termination with copies of the certs
Multiple web servers (e.g., Apache2) running HTTP
Right now we register/renew the certificates on our one HAProxy servers and then copy / cat the crt/key/ca-bundle to be used by HAProxy (using a cert-list). And also copy the certs to the other HAProxy for emergency failover.
I'm hoping there is an easier way to do this that might remove the certbot from the HAProxy server(s). For example, a different server that runs certbot and a better way to make the certs available to the proxy servers rather than a manual script to cat/scp.
Thank you for your time, and any ideas are welcome!
How do you synchronize the HAProxy configuration between the multiple instances?
From a special point of view, the certificates with their keys could be considered as part of the HAProxy configuration to be replicated.
Thanks for your reply. We don't automatically sync the HAProxy configs. They very rarely change, so we just "copy & paste" to synchronize them.
Is it frowned upon to scp the certs between nodes instead of each server generating their own? I know from a compromise-perspective it's a little bit less secure.
I was wondering if it would work to have a separate instance ("cert server") that would manage the certs and run the cert bot. HAProxy would have a forward/backend to send requests for the .well-known/acme-challenge URLs to the cert server. Then the cert server would scp the cat'd .pem files to the proxy servers after registration or renewal.
Does that make sense, or is it a bad idea? The scp part feels dirty for some reason...
Yes, that's a fine approach. A similar thing is useful behind load balancers, for example. It is better to copy them instead of having each server make their own - that is duplicative (wasteful of LE resources) and could more quickly run into rate limits.
It is good idea to have a central dedicated server to generate all the certificates. I am doing the same thing, and copying the certificates to multiple places with their own keys.
I am doing for example for BigIP F5, and SMTP/IMAP servers, and so on.
If you don't change the private key (not taking sides on that issue - just saying "if"), then you could easily script the retrieval of the matching public certs (and their chains) via many public methods.
I don't use HAProxy myself but a technique I quite like for multi-server cert distribution is to put each device in charge in charge of fetching it's own cert(s) from a secrets vault, this decouples the cert distribution from being tied to specific servers (e.g. scp from a source) and the device/client only has it's own credentials and can only fetch it's own cert(s).
We were able to tar.gz the letsencrypt stuff over to the new server, and get renewals (--dry-run) working through the HAProxies.
But I can't figure out the best way to sync the certs back to the HAProxy servers. I see that HAProxy has an "ssl crt" option where you can specify a directory, but all of the cert files in LE are in their own subdirectories by domain, and HAProxy doesn't seen to scan the subdirectories of the path I specify in the "ssl cert" configuration.
I'm exactly trying to get the ssl crt directory option working, but it doesn't seem to scan the subdirectories, so I'm not sure the best way to scp/rsync the LE certs from the certbot server over to the HAProxy servers.
I found this post which seems to indicate that I'll have to merge the files in order for them to be read by HAProxy even with the ssl crt directory feature.