Backend webservers

I have a linux virtual server LVS tunneling with a load balancer this attend the client petitions and It redirect to the backend apache servers. All the web servers resolve the same URL address.

My questions are:

Need I install the certbot only in a web server ? Or can I run certbot in every server, can Lets encrypt run a different certificate in every web server.

If i only need run a certbot in a one server ¿What files i need to copy ?

I have a nfs server, i can use it.

How can I copy immediately after certbot runs, can I add lines in the certbot script inside the /etc/cron.d/certbot

After [re]reading your post, I think you may do well by obtaining either:

  • a wildcard certificate [if each server will be able to be reached via the common name and also via a unique FQDN]

  • a single certificate that covers the common name [that can be obtained via a dedicated system and shared/deployed to each web server]

This type of setup/configuration has been covered here a few times before.
There is no need to reinvent this wheel.

1 Like

The simplest method is to redirect all /.well-known/acme-challenge/ requests to another FQDN.
That other FQDN can resolve to a single IP [not load balanced].
That system can then easily obtain certs for the redirected name via HTTP [no DNS APIs required].
Then it's a simple matter of sharing the cert and having jobs on each web server to pickup the latest cert periodically OR running a job to update all the web servers with the new cert each time the cert is renewed.

3 Likes

What kind of load balancer are you using?

This is extremely different for layer 3 and layer 7 load balancers. Main difference being where you terminate TLS.

4 Likes

@9peppe, even if the load balancer terminates the TLS, they may still want a cert for other reasons.
Now... that wouldn't explain why they would want the same named cert on all the backend webservers.
So, I'm expecting this load balancer to be layer 3 only.

2 Likes

Could be, I don't know. Sounds unusual for a stateful protocol.

4 Likes

A layer 4 using tunneling, the load balancer decapsulates the packet to the web server and the web server replies directly to the client. it does not use a nat resolution, the webservers mount from a nfs server the webpage( /var/www) they set up the identical web page, if one goes down the other web servers continue working responding AND resolving the same domain, Is Virtual IP for the same domain, all the web server have a public ip

1 Like

In this setup, how can the replies all match the same server IP [on the way back]?

As I understand it:

Original packet:

  • source: x.x.x.x:x
  • dest: y.y.y.y:y
    [where y.y.y.y is the VIP and :y is the load balanced port (80 or 443]

Reply packet:

  • dest: x.x.x.x:x [this part will always match]
  • sources: y.y.y.1:y OR y.y.y.2:y OR y.y.y.3:y [this part will never match]
    [where .1, .2, .3 are the real IPs on the backend web servers]
2 Likes

Thank you by your help.

About Reply packet:

  • dest: x.x.x.x:x [this part will always match]
  • sources: y.y.y.1:y OR y.y.y.2:y OR y.y.y.3:y [this part will never match]
    [where .1, .2, .3 are the real IPs on the backend web servers]

The sources (webservers have a tunnel interface with a virtual IP) this have a public IP to responde directly.

LVS-Tun is an LVS original. It is based on LVS-DR. The LVS code encapsulates the original packet (CIP->VIP) inside an ipip packet of DIP->RIP, which is then put into the OUTPUT chain, where it is routed to the realserver. (There is no tunl0 device on the director; ip_vs() does its own encapsulation and doesn't use the standard kernel ipip code.The realserver receives the packet on a tunl0 device ( and decapsulates the ipip packet, revealing the original CIP->VIP packet.

lvs tunnel

At this moment I am using SECTIGO certificates. These certificates are stored in an nfs to be mounted on each web server.

Could I run certbot on one webserver and then export the entire /etc/letsencrypt directory over nfs to the rest of the webservers? Would you recommend this? Or what do you propose?

I have only used letsencrypt on a server but never on a cluster. Thanks again for your valuable contribution.

1 Like

Yes.

Not really [that's overkill and could lead to confusion down the road].

Only copy the latest cert files [found via the /live/ symlinks] to the nfs share.

3 Likes

I'm finally understanding.

Only copy the latest cert files [found via the /live/ symlinks] to the nfs share.
New questions about your answer:

Do I need to install the lets encrypt software on all servers if this is the case?

Would I just have to run the cerbot to generate the certificates on a single web server and this server do nfs to export the /etc/letsencrypt/live/ folder which is the one that contains the certificates.

This would then help to avoid problems when they are renewed every 60 days (keep them in sync). This way you would no longer need to add any script or lines to the cerbot that runs under cron.d

No, the certs are just files. Once you have them just copy around like any other. Of course, the private key is sensitive info so you must keep that secure (the privkey.pem file in Certbot).

Yes, pretty much.

Certbot has a --deploy-hook which can run a script to do something with the files when they are updated. Or, create your own script for the cerbot renew and run that in cron instead.

Certs are usually renewed with 1/3 of their life left so right now that's with 30 days remaning. You don't have to instantly distribute the updated cert files and reload your servers. You could do this at a set time per day or whatever.

4 Likes

I have doubts about this file options-ssl-apache.conf , it is not necessary on the rest of the servers?.

The cert is the cert.
The way you use a cert is whole other thing.
Which protocols you chose to use, which ciphers you chose to use, etc. are not dictated by the cert nor the ACME client.
That "options-ssl-apache.conf" file is a general use case suggestion that was provided by certbot when you had it make the HTTPS vhost for you.
ACME clients are no substitute for web server administration.
The web server admin should know how to configure those options.
The only thing required outside of that are the two cert files [fullchain.pem and privkey.pem].
[everything else is NOT a requirement]

3 Likes

You have an advanced server config I would think you would want better control of the config. Updating the cert files and doing a graceful reload is one thing. Updating a file that controls your TLS config is very different.

You can certainly use the suggested options from Certbot in that include file to start. But, you should take care about applying any changes to it to ensure it still meets the needs of your servers and connecting clients. Personally I prefer managing those myself.

Certbot uses the suggestions from Mozilla. See the Mozilla configurator below. I recommend against Stapling and HSTS Strict Transport until you know what they do exactly. They can cause trouble.

4 Likes

For multi-server configurations using the same cert my suggested strategy is:

  • Make one server in charge of renewals, it doesn't strictly need to be one of the webservers
  • Use DNS validation instead of http challenges, that way you don't need to coordinate the http challenge response on all of the servers.
    • Alternatively it's still possible to use http challenges if you write the challenge response to a shared volume and make that accessible at /well-known/acme-challenge on all responding servers.
    • Yet another solution is to direct all /.well-known/acme-challenge requests to a single server.
  • Consider using a secrets store (hashicorp vault etc) for your cert full chain file and key file, this way your authorized servers can just pull the latest version of the files as regularly as they like. If the files have changed you should then move the new files to your individual servers certificate path then reload your server configuration to pick up the latest certificate. Using a file share for certs is ok but you'd still need something to signal that the configuration should be reloaded.

Ultimately, certificates are just comprised of the public key/certificate and the private key that was used to request the certificate from the CA. These are commonly PEM encoded text files (or alternatively a .p12/pfx archive) that can just be copied around, but each service that needs them must known where to look and must know to reload them when they change.

Acquiring a certificate (using an ACME client) can be completely disconnected from deployment (applying the cert to your services), but the most common usage is to acquire and deploy on the same machine which is use case certbot etc are optimised for.

4 Likes

I'm not sure if I understand what you are doing, but maybe this will help.
I have multiple subdomains, a.domain.org, b.domain.org, etc., handled by
different Apache servers, all routed by Nginx as a front-end. I have one
wildcard certificate for *.domain.org installed on Nginx. That's it, it handles
all of the servers for the different subdomains.

Thank you all.

It was easy with your help to solve my problem. I left only one web server working with the load balancer, then I installed the certificate on that web server that serves the requests directed to my domain name. Then I made that same web server nfs to mount the certificates on the rest of the servers only the files (fullchain.pem and privkey.pem), I am studying to build a script that restarts Apache to load the new certificate once it is updated.

If anyone has a script as an example to synchronize files or restart Apaches after it is updated by cerbox and can share it, it would be very helpful.

1 Like

You can add systemctl restart apache2.service (or similar, just use what you would type at a root prompt) as a deploy hook.

2 Likes

That only works on the server that ran Certbot. And, a reload is less disruptive

They could schedule a standard graceful reload of Apache daily which would pick up any fresh cert. Of course fancier ways are possible too :slight_smile:

4 Likes