Cert/s for pfsense load balanced servers

I’ve been searching to solve this problem for two days now and simply cannot so it’s time to ask for help.

I have a domain, let’s call it www.domain.com whose DNS A record points to a pfsense firewall.
On the firewall, I have two web servers set up in a load balancing configuration.

The load balancing works fine but there is something I am simply not understanding in terms of generating my certs for this kind of setup.

The servers are respectively called lb01.domain.com and lb02.domain.com.
On each, I ran the following.

certbot -d *.domain.com -d domain.com --manual --preferred-challenges dns certonly --server https://acme-v02.api.letsencrypt.org/directory

I did this because I also tried
certbot -d *.domain.com -d lb01.domain.com --manual --preferred-challenges dns certonly --server https://acme-v02.api.letsencrypt.org/directory

And the above didn’t work so I read and read and found someone talking about this so tried their idea. Nothing. When I test from SSL testing sites, I get a mismatch.

DNS:lb02.domain.com
Total number of SANs: 1

or

DNS:lb01.domain.com
Total number of SANs: 1

So obviously, I’m not generating the SSL certs correctly.

I can see my test traffic getting to the server/s but SSL testing shows mismatch.
What am I missing to make this work? Do I need to copy the first lb01.domain.com certs to any other servers, in this case, lb02.domain.com?

Hi @Lorance

what didn’t work? Please answer the following questions. Your domain name, the output of the commands etc.


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):


Hi, thanks for your reply.

My customers domain name in this case is logimon.app. There won’t be anything to see since it’s an app that will be running behind this setup which is not yet installed.

I ran this command:

I did show what I’ve run.
The setup is two web servers load balanced by a pfsense appliance.

I cannot seem to confirm what I need to use when wanting to generate a wildcard cert nor is it clear if I need the exact same cert on each server or if they each need their own.

certbot -d *.logimon.app -d logimon.app --manual --preferred-challenges dns certonly --server https://acme-v02.api.letsencrypt.org/directory

After running this, I then ran

certbot
–authenticator standalone
–installer apache
–pre-hook “systemctl stop httpd.service”
–post-hook “systemctl start httpd.service”
–server https://acme-v02.api.letsencrypt.org/directory

Everything seemed to install and testing to each server from an SSL test site showed each as working perfectly.
At that point, I had lb01.logimon.app and lb02.logimon.app in the DNS records.

This works fine IF I wanted to have two separate web servers maybe using round robin but I need to have www.logimon.app as the public facing domain name and not the individual servers. It is also not clear how I will deal with cert updates if those individual DNS server entries are removed but I suspect they will not want them in the DNS unless they absolutely must be there to make this work.

Apache 2.4.6
Centos 7.6
Self hosted
Root access, no panel
certbot 0.35.1

I don’t understand your setup.

Checked that domain there are some certificates ( https://check-your-website.server-daten.de/?q=logimon.app#ct-logs ):

Issuer not before not after Domain names LE-Duplicate next LE
Let’s Encrypt Authority X3 2019-08-03 2019-11-01 lb02.logimon.app - 1 entries duplicate nr. 1
Let’s Encrypt Authority X3 2019-08-03 2019-11-01 lb01.logimon.app - 1 entries duplicate nr. 1
Let’s Encrypt Authority X3 2019-08-03 2019-11-01 *.logimon.app, logimon.app - 2 entries duplicate nr. 2
Let’s Encrypt Authority X3 2019-08-02 2019-10-31 *.logimon.app, logimon.app - 2 entries duplicate nr. 1

But your standard urls use wrong certificates:

Domainname Http-Status redirect Sec. G
http://logimon.app/
184.164.141.219 200 0.297 H
http://www.logimon.app/
184.164.141.219 200 0.303 H
https://logimon.app/
184.164.141.219 200 5.053 N
Certificate error: RemoteCertificateNameMismatch
https://www.logimon.app/
184.164.141.219 200 4.210 N
Certificate error: RemoteCertificateNameMismatch

Your connections:

Your non-www uses the lb01, your www uses the lb02 certificate, both are wrong.

Looks like your setup is too complicated. The only thing you need is one certificate with *.logimon.app + logimon.app. You can use that certificate internal with your non-www and your www-domain and with both loadbalancers.

So you have already one correct wildcard certificate. Then use it.

I’m sorry but I’m not clear on your reply.

What do you think is complicated? The only thing I need help with is the SSL part. I’ve never set this kind of thing up using load balanced servers. Separate servers are easy.

Two servers, lb01.logimon.app, lb02.logimon.app behind a pfsense load balancer and an entry of www.logimon.app.

As mentioned, from the info I found, it was not clear what I should be doing.

The only thing you need is one certificate with *.logimon.app + logimon.app

Sorry, I’m a little confused. This is what I did no? Am I missing something here?

certbot -d *.logimon.app -d logimon.app --manual --preferred-challenges dns certonly --server https://acme-v02.api.letsencrypt.org/directory

No need to generate certs for the server names, lb01 and lb02 so I won’t do that again.

So you have already one correct wildcard certificate. Then use it.

Yes, I asked this. Should I be copying the certs from one server to the other then?
Would it not be better if I could run the commands on each server so that I could automate this renewal otherwise, have to remind someone to do this manually every so often.

PS, your test shows lb01.logimon.app, lb02.logimon.app but those DNS records were removed once the certificates were installed using the http method originally. Either this came from the servers or DNS was not fully propagated yet.

That’s the easiest solution. Create one certificate and copy it, so every instance is able to use that certificate.

You can do that. But there are rate limits.

If you create one certificate and copy it, that’s not a rate limit problem.

These are the domain names in the certificates used, not DNS entries.

Thanks for the clarifications. I’ll try to follow what you’ve told me and update my results.
As you said, rate limit should not be a problem since there are only two servers. I’ve read that you can ask for extensions if you have a fair case.

Thanks.

If I understood correctly, this is what I’ve done and now it seems to work.

First, I removed the lb01 and lb02 entries in the httpd.conf file and made them both www.logimon.app and logimon.app only.

I re-installed on both servers as follows;
certbot -d www.logimon.app -d logimon.app --manual --preferred-challenges dns certonly --server https://acme-v02.api.letsencrypt.org/directory

I then ran
certbot
–authenticator standalone
–installer apache
–pre-hook “systemctl stop httpd.service”
–post-hook “systemctl start httpd.service”
–server https://acme-v02.api.letsencrypt.org/directory

I was asked if I wanted to re-install the certs or install new and I went with re-install.
A couple of glitches later, it worked once I cleaned up some left over stuff in the apache config from the previous tries.

Testing on SSL sites, now I see a constant return for www.logimon.app even as the hits go from one server to the other but not for logimon.app. Still missing something.

I’d love to use this in some other applications so it would be nice to know what I’m missing to have both www and logimon.app. I thought the extra -d logimon.app would take care of both.

I think I know where I made an error. Will test this now.

1 Like

Finally found the problem and it’s when I was being prompted while installing and restarting the web server. I was picking an option rather than allowing bothdomain and sub domain.

I also removed the old entries using certbot delete -d domain-names

Seems to be working now :).

Do the well known challenge DNS records have to remain in place? I still see something remote trying to reach those.
Last but not least, I’ll need to know how to renew these automatically so as not to cause any down time.

1 Like

What’s that?

If you use dns-validation, there is no /.well-known/acme-challenge subdirectory used.

If you use http-validation, there is no special DNS record to create.

And I don’t understand your setup.

There you use dns-validation.

There you have a standalone, so you use http-validation.

Three certificates? Two different challenge types?

If you use --manual, you can’t automate that. Dns-validation -> your dns providerr must support an API, Certbot must support that API. But if you want to create a wildcard certificate, dns-validation is required.

Downtime has nothing to do with certificate creating. Only thing: Certificates are 90 days valid. So you have to create new certificates early enough, Certbot typically renews certificates after 60 days. So if there is an error, it’s time enough to fix that.

You’ll want to pick one of your servers as the main issuing server and set it up with a cron job to renew (certbot may already have installed one of these). You’ll also need to set up a role account on the other server that has appropriate permissions to install a certificate. Then configure a cron job on your issuing machine to SCP certificates over to the other one and (this is important) reload your web server via SSH.

There are only two servers involved in the load balancing. Could I not just automate it on each? Having to copy is an extra step that could end up being missed when the time comes.

Yep, you can automate it on each! Keep in mind that that will count as 2 duplicate issuances towards your limit of 5 duplicate issuances per week, so there’s a limit to how many LBs you can have doing that at once.

Also, separate automation for each frontend is challenging when using HTTP-01 challenges, but since you’re using DNS-01 challenges, it should be pretty straightforward. The main thing to make sure of is that client 1 doesn’t delete TXT records provisioned by client 2, if they’re both running at the same time.

Seems I need to learn more about how letsencrypt works then. I thought it was just plain free to use for a fairly large number of certs. I’ve not once read about a limit of 5 but I’ve seen people talking about hundreds and being able to ask for extensions.

There are just too many moving parts to keep track of this in a micromanaged way which is why I was thinking of just letting each server automatically update its own cert when needed.

I was considering setting up another couple sets of load balanced servers but now I’m just confused.

There’s a big list of the different rate limits:

You can get hundreds of certificates, but you can’t easily get hundreds of identical certificates.

If you want to have 6 different load balancers, you could avoid the duplicate certificate rate limit by making 1 (or more) certificates and copying them around, or by adding a useless subdomain so that they’re no longer duplicates. E.g.:

  • Server 1: example.com, www.example.com, server1.example.com
  • Server 2: example.com, www.example.com, server2.example.com

You can request a rate limit increase – a popular example is a university with thousands of different subdomains independently managed by dozens of different people – but I’m not sure you can get an increase to the duplicate certificate limit – easily, or at all – since it’s usually easy to work around.

Thanks for the help and input everyone. I’m now using letsencrpyt more than the usual SSL cert providers.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.