Renew certificates behind load balancer (IIS)

Hi,

We have 2 servers running IIS behind a load balancer, and those have our website published under the same DNS name: secure3.mydomain.com.

I was able to use Win-acme to generate a Let's Encrypt certificate on Server 1 and now I need to have that same certificate (or any other for the same domain name) on Server 2.

I believe that if I do the same procedure on Server 2, it's not going to work because when Let's Encrypt tries to validate the HTTP URL it's going to find Server 1 instead.

Is there any way to have this set up for both servers without having to change the website URL on any of those?

Thanks,

Carlos

2 Likes

The best option is to install the certificate from Server1 onto Server2.

To deal with Load Balancers, the EASIEST setup is to do one of the following:

  • proxy all traffic for /.well-known/acme-challenge onto one server
  • redirect all traffic to /.well-known/acme-challenge onto a hostname that only one server will respond to

On a simple setup, you can have a Server1 share the certificates via a network file share. On a more advanced setup, you can copy them over using post certificate hooks.

There are a lot of options, depending on how complicated or easy your stack is. The general idea is you want to procure only one SSL Certificate from LetsEncrypt, and share it across your nodes.

5 Likes

And maybe the load balancer can terminate TLS connections itself? So the load balancer would get and use the certificate.

7 Likes

My preference for this scenario (and the one I recommend to Certify The Web users on IIS) is to use DNS validation instead of http validation, that way you don't have to forward http challenge requests to an authoritative server (the one running win-acme in this case).

If you then use the Windows Central Certificate Store feature for your IIS https bindings then all your IIS servers can share the same set of certificates and they will all use the latest cert. You either host a UNC share they can all see or you copy the cert multiple times to a share on each server during your deployment after your renewal. Certify happens to have a Deployment Task feature so you can execute multiple tasks when your cert renews and you can conditionally run tasks on success/failure (like notifications etc).

3 Likes

NOTE: This post is basically a "what if sidetrack"; Read or respond at your own discretion (as always).

I like this idea most. But in the event that anyone might think it would leave the servers without certificates and thus when connecting to them while behind the load-balancer (if that is even your case), it would only be able to use HTTP...
To overcome that: One could always use a domain certificate (if domain joined), OR any other cert type, within the private network side of the load-balancer.
But most would agree that keeping the servers behind the load-balancer at all times is likely a better condition.
That said, it would appear to leave the connections from the load-balancer to the servers unencrypted.
So, for those "truly paranoid" (myself included): Any cert type could be used there. A single, very long-lived, cert could be used on all the servers; And trusted by the load-balancer.

Sorry about the length of this sidetrack... in this topic.
We now return you to your regularly scheduled program topic discussion.

2 Likes

Hi everyone,
Thanks for all the answers.
I believe that using the certificate in the Load Balancer would be a better solution, but it'll require much more effort from us at the moment.
I'm more to have certificates on each of the servers, even if it's needed some manual work during the renewal time.
Today morning, I was able to generate a certificate using HTTP-01 challenge and also schedule its renewal.
Initially, I tried opening port 80 to the Load Balancer, but it hasn't worked.
Then, I ended up opening port 80 directly to Server 1 and everything worked fine.
Let's Encrypt certificates have been stored in the Web Hosting store, while our previous certificates were in the Personal store, but it seems it works fine the way it is.
Then, I tried exporting the certificates to be able to import those on Server 2.
The export wizard hasn't allowed me to export the private key with the certificate, which limited the formats I was able to use to 3.
I tried 2 of those and imported the exported file to both Personal and Web Hosting stores on Server 2.
Next, I opened IIS and edited the bindings for our website.
When I opened the SSL Certificate combo box, none of the imported certificates were there.
I'm currently stuck at this point.
I was considering then, doing the exact same procedure I did for Server 1 to server 2 (changing the firewall configuration).
Then, if I run win-acme on Server 2, it's going to create a certificate for that server.
My concern is that Let's Encrypt detects that the domain name for Server 2 is the same as the one previously created for Server 1, and instead of creating a 2nd certificate for the same URL, it creates that and somehow makes the one installed on Server 1 to stop working.
Does anyone know what it'd happen in this case?
Also, is there any way to export the Let's Encrypt certificate from Server 1 and import that on Server 2, in a way that it'd be available for me to use that on IIS?
Thanks,
Carlos

2 Likes

A usable Let's Encrypt certificate consists of a public and private key pair, on Windows this is conventionally packaged in a PFX file (also known as a PKCS12 container format, or .p12), on other operating systems and services these are often split into a few different files. There is nothing special about the certificate itself which stops you working with it or converting it in various ways, but there is something specific about the way the cert and private key have been imported into the (protected) certificate store.

win-acme will output a bunch of files and as part of the default process it will attempt to store the PFX in the local machine certificate store, under Web Hosting (this is it's default, the Personal/My store is also fine, it just has to be under Local Machine, not a specific user). It will use a random password by default so you will probably want to specify one instead.

Your process is falling down because you are manually exporting the certificate and losing some information. If you can't export your private key then you cannot use the certificate (it's just a public certificate/key at that point, not one you can use).

I'd suggest you use the win-acme option to export to a PFX file (PFX File Plugin) and set the export location and password win-acme alternatively mark the private key as exportable in your win-acme settings:

For general win-acme help you should use their GitHub discussions: Discussions · win-acme/win-acme · GitHub

4 Likes

Thanks for the explanation and the valuable tips webprofusion.
I was able to create new certificates on Server 1, and I believe those will auto-renew accordingly from now on.
I did that running Win-Acme and following the menus in the application. One thing that surprised me is that it still asked me some questions about the destination folder and password to be used for the pfx file, even though those were already set up in the settings.json.
I expect that when it renews, it creates new .pfx files in the same folder with no human intervention.
So, this solves my problem that was to replace paid server certificates with free ones.
I was also was able to successfully import those to Server 2 and it seems to be working fine.
Now, I'm planning to improve this setup by adding as much automation as possible, so I won't need to do a lot of manual stuff every 55 days.
My idea is to create a script to import the certificate to Server 2 (after each renewal) and change the website binding on IIS to use that new certificate, instead of doing it manually.
I'm currently researching on CERTUTIL and trying to see if can do that, but I have no clue on how to change an IIS website binding through a script yet.
Did anyone had to do something like that before?

2 Likes

As I previously mentioned, one of the simplest options for deployment is just to use Windows Central Certificate Store (CCS). You would need copy your certificate to the machine with the file name as www.yourdomain.com.pfx (one file for each domain or subdomain, or _.yourdomain.com.pfx for wildcards), then configured the CCS feature in IIS and edit your IIS https binding to use CCS. The server will then just use whichever matching certificate file it finds in that folder.

win-acme has a built in option to copy the file with the correct naming: win-acme (as does Certify The Web Deployment Tasks | Certify The Web Docs).

4 Likes

Hi Webprofusion,
It seems really a great setup.
I was researching about that and then imagined the following scenario.
Suppose that I have CCS on our Management Server and both Web Server 1 and Web Server 2 use the certificate from that.
Let's say I need to restart the Management Server and it takes 1 hour to get back.
Do you know if those Web Servers (1 and 2) would stop working?
Or does it depend on the CCS just to select the certificate from a list and cache that on the Web Server itself?
I'm asking that because if that depends on the CCS server to be online for the Web Servers to work, I'd try to create a file server cluster between the 2 Web Servers and put the CCS there.
Otherwise, if the Web Servers just need to access the CCS in the time when we select which SSL certificate the server will use, I'd create the CCS on the Management Server.
Thanks again! Carlos

3 Likes

If your share disappears then you could expect that your websites would stop working yes (Thoughts on the Central Certificate Store Feature in Windows Server 2012 - Microsoft Tech Community).

You could just have a share on each web server and copy to those instead, that way it's always local but your still managing to avoid updating the machine certificate store and IIS bindings.

Often though load balanced webservers are using shared storage for their web application files etc (via a SAN) so if you are using a network attached file system you could just copy once.

2 Likes

On extreme traffic servers, best practice is the certificate on the front balancer, copies of the certificate can be also placed on each of the traffic servers. This can scale all the way to 1 million page views per hour which is what Facebook has to deal with.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.