Lets Encrypt Certificate on 2 Apache Servers via Round Robin

I have a private Website running on a (Web)Server which is accessed via a Reverse Proxy (Apache2) For Testing and other purposes I have an identical 2nd (Web)Server in a different location also accessed via another Reverse Proxy (Apache2)

Both Reverse Proxy Serves running Apache have a unique IP4. So now I would like to add both IP4s to my Domain located at my Domain Hoster (both as A-Record) as Round-Robin

Because I am bad with words - here is a picture

My issue is, that I do not really know how to set up the SSL Certification via Let's Encrypt, so that both servers are having the same certificate. (Currently, the first ReverseProxy as done a normal HTTPs Challenge and created the certificates and I was afraid to run the certbot on the 2nd server again and making the certificate of server1 invalid)

Constraints: I do not want to manually copy the certificate files from ReverseProxy to another.

1 Like

Hi @cmasterdelphi

normally, that can't work.

Because if your client runs on Proxy1 (or Webserver 1), then a validation file is created. But if Letsencrypt checks Proxy2 / Webserver 2, the validation would fail.

Read

So you have two options:

  • Use dns validation
  • Use a redirect proxy1 -> subdomain1, proxy2 -> subdomain2, but subdomain1 / subdomain2 has always only one ip address. Then you can run Certbot1 with the webroot of subdomain1 on proxy1 and Certbot2 with the webroot of subdomain2 on proxy 2.

That problem doesn't exist. You can create max. 5 identical certificates per week.

2 Likes

With only 2 endpoints, I'd probably setup an ACME client on each reverse proxy box such that they both get functionally identical certificates independent of each other. You won't likely run afoul of any rate limits and you can even stagger the renewals such that if something goes wrong, only one of the two would ever be affected. The only tricky part with this setup is dealing with the challenge validation.

If you want to use HTTP challenges, you'll have to set up some way to sync the expected challenge files onto both systems because you won't be able to guarantee which proxy the requests hit.

Using DNS challenges is much less hassle assuming your client has a plugin for your DNS provider.

3 Likes

It doesn't make it invalid! Web PKI certificates don't contradict each other; each one says "this is OK" but not "this is the only OK thing".

But as other people have mentioned, getting the validation from the CA to succeed in this situation can be difficult if you use the HTTP challenge.

A somewhat complicated workaround could be to have alternative distinctive DNS names for these machines (not the same names that you're trying to get the certificate for), and configure the reverse proxy on each machine to forward requests for http://thesharedname.example.com/.well-known/acme-challenge/ to the other machine using its distinctive name, e.g. rp1.example.com redirects http://thesharedname.example.com/.well-known/acme-challenge/ to http://rp2.example.com/.well-known/acme-challenge/, while rp2.example.com redirects http://thesharedname.example.com/.well-known/acme-challenge/ to http://rp1.example.com/.well-known/acme-challenge/.

Then make your Certbot pre-hook script on each machine temporarily disable this forwarding behavior on that machine, and make post-hook re-enable it.

In case you get the certificates at not-exactly-identical times, each machine would then cooperate in making the CA's validation requests go to the other machine, except when it expects to receive them itself.

This is probably much more complicated than necessary—in particular, it's probably much more complicated than using something like the DNS challenge method instead. :slight_smile:

1 Like

I was thinking you could also potentially take advantage of cached authorizations as a solution to HTTP challenges. Do a one-time sync so that both servers are using the same ACME account with their instance of the client. Configure the reverse proxy so all of the challenge requests go to server1 and have that one renew first. Then the second server won't need to do a new validation as long as it renews within 30 days of the first.

Again though, DNS challenges would be way simpler.

3 Likes

@rmbolger, that's clever. It's also a great example of something that "works in practice, but not in theory" (the ACME protocol doesn't guarantee that clients can do this, but the current functionality of the Let's Encrypt service would make it work reliably).

2 Likes

Authz aren't cached for 30 days, but just 7 AFAIK.

1 Like

The FAQ says 30 days at the moment. Did that change recently?

1 Like

Hm, I didn't get that memo, always thought it was 7 days :stuck_out_tongue:

Seems to be 30 days since at least 2017, I'm probably confused with the CAA validity.

2 Likes

@Osiris, you are not alone, I thought it was 7 days too :crazy_face:

1 Like

Thank you all for your replies and hints. I am just going through all the options.
The DNS-01 challenge seems to fit the best, but I need to figure out if this works with my Mail-in-a-Box Setup. (I notice, I have a quite complex setup for some private stuff, but it was the cheapest option for my needs)

Does anybody have experience with Mail-in-a-Box and the DNS-01 challenge? Otherwise I might need to look closer to this redirect solution.

If you use the dns challenge, your local configuration isn't really relevant.

  • Outbound connections are always required, your ACME-client must be able to connect Letsencrypt
  • Your dns provider should support an API you can use. If not, you have to use --manual. Always possible, but painful, no automation. What's your domain name / your name servers?
1 Like

Well, didn't mentioned in the first post, because I didn't know that it was necessary and thought it would add too much complexity.

My Domain Hoster actually sends everything to my Mail-in-a-Box, which acts again as some kind of reverse proxy but not really. There I configured certain custom entries which forwards all the request to my IPs (with the next reverse proxy) pushing it further to my webservers. The obtaining of the lets encrypt certificates is currently done by the apache2 reverse proxies (they actually do nothing else but to distribute the requests between the webserver and another storage unite, not drawn in the picture)

I hope this picture explains my situation properly.

//Edit: I started the question about the DNS challange at the mail-in-a-box discussion forum.

that sounds promising, since I might fail to automate the dns-01 challenge. How do I configure the certbot to use the same "ACME" account? So far I only ran the command "certbot --apache -d domain.tld" in order to create the certificate.

The way I have addressed this in the past:

Option A) in-network setup
Network A runs LetsEncrypt; Network B has ".well-known" proxied into a rule on Network A

Option B) off-network setup
Office Computer - runs letsencrypt; custom script places the challenge into NetworkA & NetworkB servers

To share the certs,

  1. nightly cron jobs sync the cert directories, or
  2. post-issuance hook
1 Like

OK, so I force renewed the certificate on proxy1. Then I changed the DNS from IP1 to IP2 and ran certbot on webserver2 for the same domains (successfully). Then I added IP1 again to the DNS (round-robin).

My question is, will the renewal now work automatically or will it fail in 90days? :sweat_smile:

To be honest, I'm not super familiar with low level certbot stuff as I'm generally using my own client. But playing around on a test machine, it might be sufficient to just tar up the contents of /etc/letsencrypt/accounts, copy it to the second machine, and untar it there.

I'm not sure if it complicates things if you've already tried getting certs on that second machine though. Someone more familiar with certbot internals would have to verify. You might want to do a certbot delete --cert-name blah for any cert orders that may exist on that machine first.

Alright. Just as documentation for my solutions:

I went with the DNS-01 challenge via the acme.sh script. | GitHub - acmesh-official/acme.sh: A pure Unix shell script implementing ACME client protocol
This provided a plugin for Mail-in-a-Box | dnsapi · acmesh-official/acme.sh Wiki · GitHub

I am not sure, if I did everything correctly so I set it up on a test page and now I will wait for 3 month to see if the renewal really works automatically.

I wanted to thank everybody, who gave me useful tips and hints.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.