Automated deployment of key/cert from reverse proxy to internal systems

I have about 20-30 subdomains being hosted on various systems (mostly Apache, some Nginx) which sit behind an Apache reverse proxy. Setting up the reverse proxy machine to obtain HTTPS certs was fairly trivial, but now comes the problem of Google and Mozilla’s changes to their web browsers which require me to have HTTPS also be available inside the network.

Is there an automated and reasonably secure way to deploy the certificates to the other systems behind the proxy?

The methods I have considered and discarded:

  • simply re-run certbot on each machine (prohibited by rate limiting, but easily the best option)
  • rsync from proxy to each system (very insecure)
  • configure an http server which contains the key/cert only available on the local network with a script on each of the other machines to copy updated keys (complicated, huge security issues)
  • somehow upload the key/cert to a different machine inside the network on renewal and somehow inform the internal servers so they can download them and re-deploy (really complicated)

I looked into doing this with Jenkins as well, but I couldn’t find something ready-made to do the job. If anyone knows of something, the help would be greatly appreciated!

Can you elaborate? Whether your reverse proxy is talking to your backend servers over HTTPS or not is invisible to the browser. As long as it sees a valid certificate, the browser is happy.

Why is that?

Certificates are public information (they're sent in every HTTPS connection, and available in public CT logs).

You only need to copy the private key to each server once when they're stood up (boostrapping), and then make sure Certbot runs with --reuse-key on the proxy.

From there, you only need to distribute non-sensitive data: the certificate.

You can also have the servers pull (rather than push) the updated certificate from the proxy.

This doesn't need to be stateful. Just have a cron job that pulls the certificate every 12 hours and does a graceful reload of the webserver. Both Apache and nginx will happily do this without interrupting traffic.

The only complicated part is having monitoring in place to ensure it isn't silently failing.

The DNS cannot be changed to point to the reverse proxy instead of the servers themselves, so the internal traffic does need to go directly from a given client to the servers inside the network.

Unfortunately, browsers which will hit the server both from inside and outside the network will try to use https, remember that it is available (from outside) and disable plain http for that server. It is possible to go into Chrome’s security settings on desktop to make it forget that https was available for a server and again allow http, but this must be done manually every time such a machine is brought back into the internal network and for each server.

Enabling rsync is problematic because it strips away the isolation of the servers from one another, which is bad for security. Giving even selective access from one system to another creates a security hole.

The difficulty in the last setup I described is the push to neutral server, then pull from the internal ones. Choreographing that is a challenge without writing some additional middleware.

I don’t think there’s that much coreographing required.

This is approximately what I’ve done in the past.

On the load balancer

Deploy hook (/root/deploy_hook.sh):

#!/usr/bin/env bash
set -e
for domain in $RENEWED_DOMAINS; do
   # Just an example, you can use any non-sensitive storage medium you want
  aws s3 cp --follow-symlinks "$RENEWED_LINEAGE/fullchain.pem" "s3://cert-storage.example.org/certs/$domain.pem"
done

Certbot invocation:

certbot-auto certonly -a apache -d example.org \
--reuse-key --deploy-hook /root/deploy-hook.sh

On the backends

12 hourly cronjob

#!/usr/bin/env bash

# Put this in crontab for every 12 hours
# Assuming Apache, and that your private key and certificate are located in
# - /etc/apache2/privkey.pem
# - /etc/apache2/fullchain.pem , respectively

set -euf -o pipefail

# Download the latest certificate to a temporarily location so we can check validity
curl -s -o /tmp/fullchain.pem https://cert-storage.example.org/certs/example.org.pem

# Verify the certificate is valid for our existing key (should be)
MOD_CRT=$(openssl x509 -noout -modulus -in /tmp/fullchain.pem | openssl md5)
MOD_KEY=$(openssl rsa -noout -modulus -in /etc/apache2/privkey.pem | openssl md5)
if [ "$MOD_CRT" != "$MOD_KEY" ]; then
  echo "Key didn't match: $MOD_CRT vs $MOD_KEY"
  exit 1
fi

# Deploy the certificate and graceful reload
echo "New certificate: $(openssl x509 -in /tmp/fullchain.pem -noout -subject -dates -issuer)"
cp /tmp/fullchain.pem /etc/apache2/fullchain.pem
apachectl -k graceful

Private keys never touch the network, no new creation of privilege. You can run Monit on the proxy or whatever existing monitoring mechanisms you have to check that the validity of the backend servers’ certificates are >14 days or whatever.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.