Certificate Issuance with Ansible and DNS Loadbalancing

Hello everybody,

we have multiple servers around the world that are loadbalanced over the world. Each IP Address represents multiple virtual loadbalancers which each require their own certificate and terminate the https connection for all requests.

We want to switch to Letsencrypt with the start of ACME API v2 and started to check how we can automatically issue new certificates. Since all servers are installed and configured automatically, we dont want to use crontab to renew certificates (as we had some issues with it lately.)

A normal DNS Record looks like this:

subdomain.example.com. IN A 1.2.3.4
subdomain.example.com. IN A 1.2.3.5
subdomain.example.com. IN A 1.2.3.6

This makes issuance with validation over TLS-01 very difficult, because the server that requested the new certificate does not automatically handle the incoming request from Letsencrypt (though this may be okay for loadbalancers behind the same public IP, we cannot handle it over DNS).

Due to the problems described above, we decided to go with DNS Validation. Since this is also required for wildcard certificates, it is the only option for us anyway.

We plan on integrating the certificate issuance with Ansible. The plan is to reissue every certificate whenever it expires (or 30 days in advance).

The current plan looks like this:

  1. Generate a new key on the regarding loadbalancer/server
  2. Create a new CSR
  3. Send the CSR over the API to Letsencrypt (and request the DNS Validation)
  4. Automatically update the DNS record for the record
    5)Request the certificates on every server (so we end with 20 different cert/key pairs)
  5. Wait until the certificate has been issue successfully
  6. Removed the DNS Record again.

However, we have to do this up to 20 times per Record

Some thoughts:

  1. Ratelimiting
    1.1) We have less than 20 subdomains/wildcard
    1.2) We have to issue the same certificate with different keys for more than 20 servers within a few minutes/hours.
  2. DNS Verification
    2.1) The same domains are handled over all servers - can we reissue the certificate for all servers without updating the DNS for each server? e.g 1 DNS Validation record for all certificates issued with this domains
  3. Ansible
    3.1) We have our own playbook for generating new keys and csrs, but is there a module that can check if a certificate is about to expire and “kick in” the reissuance of all certificates
    3.2) if Something like above does not exist, can we (in theory :wink: ) reissue the certificates every day/every ansible run?

Thanks to everybody

This doesn't seem necessary. Is it really?
Maybe you could stager them with just a few per day.
Recall the existing cert will not be revoked, so it will still be good for the remainder of that issuance.

If you can stagger the initial issuance over a few weeks (e.g. you still have a commercial certificate to lean on or you can wait to turn SSL on for your site) you wouldn't have to stagger renewals due to the renewal exemption, as long as you added a unique domain name to each certificate to avoid the Duplicate Certificate limit.

Or you could ask for a rate limit exemption. There's a form and everything you need to qualify linked from:

If you share the same ACME account key, it should cache the validations as long as all the same domain names are used on all the servers.

If you do not share the ACME account key you would need to upload a different record for each account/domain.

Recent versions of Ansible have support for checking certificate expiry (and even obtaining certificates from Let's Encrypt!)

http://docs.ansible.com/ansible/latest/openssl_certificate_module.html

Ordinarily you couldn't do this due to rate limits. Even with a rate limit exemption it wouldn't be very polite to everyone else: each certificate takes up OCSP signing capacity for 90 days regardless if you stop using it next day.

Well we have 20 servers that do loadbalancing (most of them are containers) - so we need a certificate per server (though it may be possible to share the key/cert, I dont really want that)

We can expand the issuance over a few hours and on different days, however we want to do it completely automatic.

Thanks for the Ansible Module - looks great on the first view and can handle the reissuance when the time runs out automatically.

Thank you all, I think with the same Account key should work for the issuance :slight_smile:

See below

My general analysis is that you have made things overly engineered

A) have one server that is a dedicated PKI server
B) Install ansible and Certbot on that server
C) Use certbots hooks to kick of an ansible playbook which is passed the cert, key and intermediates
D) Install of those on each server

Some of the challenges you may run in to using your approach

A) Mapping certificates to servers and forgetting to renew a certificate
B) Managing 20 installs of certbot (keeping them all up to date)
C) managing 20 instances of the filesystem
D) Scheduling renewals
E) you may run in to trouble with HPKP if you decide to implement this later
F) Distributing you DNS api key (though you may do this with ansible) to 20 servers
G) Troubleshooting errors (not the same cert, is the cert valid is the key valid etc)

I think the pattern of using Ansible to manage lots of certbot clients on a server is a wrong approach but most of the playbooks seem to favour this (and so do articles)

A much more elegant solution and the one I have implemented for those who have asked me to integrate with Ansible is one central source of truth for certificates and use of Ansible for configuration management and deployment (which is what it is actually designed to do :wink: )

I.e. Certbot -> Ansible -> 20x Servers not Ansible -> Server -> Certbot

Andrei

also while you are testing i suggest using the --staging flags so you don’t run in to rate limits while evaluating approaches

you can then use the production servers when you have things figured out

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.