I’ve been searching to solve this problem for two days now and simply cannot so it’s time to ask for help.
I have a domain, let’s call it www.domain.com whose DNS A record points to a pfsense firewall.
On the firewall, I have two web servers set up in a load balancing configuration.
The load balancing works fine but there is something I am simply not understanding in terms of generating my certs for this kind of setup.
And the above didn’t work so I read and read and found someone talking about this so tried their idea. Nothing. When I test from SSL testing sites, I get a mismatch.
DNS:lb02.domain.com
Total number of SANs: 1
or
DNS:lb01.domain.com
Total number of SANs: 1
So obviously, I’m not generating the SSL certs correctly.
I can see my test traffic getting to the server/s but SSL testing shows mismatch.
What am I missing to make this work? Do I need to copy the first lb01.domain.com certs to any other servers, in this case, lb02.domain.com?
what didn't work? Please answer the following questions. Your domain name, the output of the commands etc.
Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is:
I ran this command:
It produced this output:
My web server is (include version):
The operating system my web server runs on is (include version):
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don't know):
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
My customers domain name in this case is logimon.app. There won't be anything to see since it's an app that will be running behind this setup which is not yet installed.
I ran this command:
I did show what I've run.
The setup is two web servers load balanced by a pfsense appliance.
I cannot seem to confirm what I need to use when wanting to generate a wildcard cert nor is it clear if I need the exact same cert on each server or if they each need their own.
Everything seemed to install and testing to each server from an SSL test site showed each as working perfectly.
At that point, I had lb01.logimon.app and lb02.logimon.app in the DNS records.
This works fine IF I wanted to have two separate web servers maybe using round robin but I need to have www.logimon.app as the public facing domain name and not the individual servers. It is also not clear how I will deal with cert updates if those individual DNS server entries are removed but I suspect they will not want them in the DNS unless they absolutely must be there to make this work.
Your non-www uses the lb01, your www uses the lb02 certificate, both are wrong.
Looks like your setup is too complicated. The only thing you need is one certificate with *.logimon.app + logimon.app. You can use that certificate internal with your non-www and your www-domain and with both loadbalancers.
So you have already one correct wildcard certificate. Then use it.
What do you think is complicated? The only thing I need help with is the SSL part. I've never set this kind of thing up using load balanced servers. Separate servers are easy.
Two servers, lb01.logimon.app, lb02.logimon.app behind a pfsense load balancer and an entry of www.logimon.app.
As mentioned, from the info I found, it was not clear what I should be doing.
The only thing you need is one certificate with *.logimon.app + logimon.app
Sorry, I'm a little confused. This is what I did no? Am I missing something here?
No need to generate certs for the server names, lb01 and lb02 so I won't do that again.
So you have already one correct wildcard certificate. Then use it.
Yes, I asked this. Should I be copying the certs from one server to the other then?
Would it not be better if I could run the commands on each server so that I could automate this renewal otherwise, have to remind someone to do this manually every so often.
PS, your test shows lb01.logimon.app, lb02.logimon.app but those DNS records were removed once the certificates were installed using the http method originally. Either this came from the servers or DNS was not fully propagated yet.
Thanks for the clarifications. I’ll try to follow what you’ve told me and update my results.
As you said, rate limit should not be a problem since there are only two servers. I’ve read that you can ask for extensions if you have a fair case.
If I understood correctly, this is what I’ve done and now it seems to work.
First, I removed the lb01 and lb02 entries in the httpd.conf file and made them both www.logimon.app and logimon.app only.
I re-installed on both servers as follows;
certbot -d www.logimon.app -d logimon.app --manual --preferred-challenges dns certonly --server https://acme-v02.api.letsencrypt.org/directory
I then ran
certbot
–authenticator standalone
–installer apache
–pre-hook “systemctl stop httpd.service”
–post-hook “systemctl start httpd.service”
–server https://acme-v02.api.letsencrypt.org/directory
I was asked if I wanted to re-install the certs or install new and I went with re-install.
A couple of glitches later, it worked once I cleaned up some left over stuff in the apache config from the previous tries.
Testing on SSL sites, now I see a constant return for www.logimon.app even as the hits go from one server to the other but not for logimon.app. Still missing something.
I’d love to use this in some other applications so it would be nice to know what I’m missing to have both www and logimon.app. I thought the extra -d logimon.app would take care of both.
I think I know where I made an error. Will test this now.
Finally found the problem and it’s when I was being prompted while installing and restarting the web server. I was picking an option rather than allowing bothdomain and sub domain.
I also removed the old entries using certbot delete -d domain-names
Seems to be working now :).
Do the well known challenge DNS records have to remain in place? I still see something remote trying to reach those.
Last but not least, I’ll need to know how to renew these automatically so as not to cause any down time.
If you use dns-validation, there is no /.well-known/acme-challenge subdirectory used.
If you use http-validation, there is no special DNS record to create.
And I don't understand your setup.
There you use dns-validation.
There you have a standalone, so you use http-validation.
Three certificates? Two different challenge types?
If you use --manual, you can't automate that. Dns-validation -> your dns providerr must support an API, Certbot must support that API. But if you want to create a wildcard certificate, dns-validation is required.
Downtime has nothing to do with certificate creating. Only thing: Certificates are 90 days valid. So you have to create new certificates early enough, Certbot typically renews certificates after 60 days. So if there is an error, it's time enough to fix that.
You'll want to pick one of your servers as the main issuing server and set it up with a cron job to renew (certbot may already have installed one of these). You'll also need to set up a role account on the other server that has appropriate permissions to install a certificate. Then configure a cron job on your issuing machine to SCP certificates over to the other one and (this is important) reload your web server via SSH.
There are only two servers involved in the load balancing. Could I not just automate it on each? Having to copy is an extra step that could end up being missed when the time comes.
Yep, you can automate it on each! Keep in mind that that will count as 2 duplicate issuances towards your limit of 5 duplicate issuances per week, so there’s a limit to how many LBs you can have doing that at once.
Also, separate automation for each frontend is challenging when using HTTP-01 challenges, but since you’re using DNS-01 challenges, it should be pretty straightforward. The main thing to make sure of is that client 1 doesn’t delete TXT records provisioned by client 2, if they’re both running at the same time.
Seems I need to learn more about how letsencrypt works then. I thought it was just plain free to use for a fairly large number of certs. I’ve not once read about a limit of 5 but I’ve seen people talking about hundreds and being able to ask for extensions.
There are just too many moving parts to keep track of this in a micromanaged way which is why I was thinking of just letting each server automatically update its own cert when needed.
I was considering setting up another couple sets of load balanced servers but now I’m just confused.
You can get hundreds of certificates, but you can't easily get hundreds of identical certificates.
If you want to have 6 different load balancers, you could avoid the duplicate certificate rate limit by making 1 (or more) certificates and copying them around, or by adding a useless subdomain so that they're no longer duplicates. E.g.:
Server 1: example.com, www.example.com, server1.example.com
Server 2: example.com, www.example.com, server2.example.com
...
You can request a rate limit increase -- a popular example is a university with thousands of different subdomains independently managed by dozens of different people -- but I'm not sure you can get an increase to the duplicate certificate limit -- easily, or at all -- since it's usually easy to work around.
