I have a server that hosts a Docker Swarm. This swarm has many services, one of which is a webapp.
There are several domains (not subdomains, although we also have subdomains) pointing to this server: webapp.domain_1.com, webapp.domain-2.com, ... So, it's our server responsibility to serve valid TLS certificates for each on of them. Up until now, we would ssh into the server, then exec a bash into the container and manually renew the files (copying the challenge file manually and so). In addition, we have to keep track of the expiration date of each TSL certificate and do all this every three months.
So, my question is, what would be a nice approach to automate this process without overloading the Let's Encrypt systems? I want to leverage the ability of LE of automatically renewing certificates as much as possible.
My first idea was to run certbot as part of the docker swarm in a service of its own. The problem with that is that as soon as the certbot command is done, the container is destroyed. There are no such thing as cron jobs per se.
Another idea was to actually run cron in a container that has certbot installed in it. But I wouldn't be able to control if I'm hitting the limitations of Let's Encrypt systems.
Yeah, I could share a volume between the "certbot container" and the rest, so that the certificates would be available for all services and domains. But how could I make it automatic?
I personally prefer the idea of storing certs in a vault (like hashicorp vault or azure keyvault) then pulling the latest on startup and on regular intervals.
Some of the container orchestration solutions have specific cert management tools/layers as well, so see if there is something specific for yours.
Definitely don't let a host that's regularly scaling down and up manage it's own certs in transient storage, because you'll hit rate limits very quickly. You could possible just have a shared volume for cert storage.
Without knowing any real specifics...
I would have to say: Try to make it "fool-proof".
Set a cron job to run on startup/reboot that sets things as needed.
Set another one to run once a week or so to keep things up-to-date.
[and definitely use a shared volume (that is always accessible) for your certs]
OR
Use high-tech software to do it for you - LOL
Well possibly, it's more something I'd consider for runtime than build time but it depends how often you expect to recycle the instances. If an instance will be up for more than 90 days (for example) you'd need a built-in certificate refresh, but if they are short lived it could just be part of the init scripting. If you are constantly re-building fresh images (then deploying them) then yes you could just include the latest cert in the image. Either way the vault approach (or a cert manager with API) avoids re-requesting much the same cert from Let's Encrypt too frequently.