Using Certbot with Immutable Web Servers - Use Dedicated Server with Configuration Management Tool

We want to follow the immutable system pattern. Therefore we do not change server configuration on a server after a server was deployed. We want to replace the server with the next version of that server (a version that in includes the desired configuration change).

With AWS EC2 this is easy. Just lunch a new server, based on a AMI (version 2), and then terminate the first server (based on AMI, version 1).

Installng software that is doing post-provisioning configuration changes on-the-fly at runtime is discuraged. In case of certbot, it should not be installed in the AMI, its account key should not exist in the AMI.

Installing certbot, running it to obtain certs, uninstall it and cleanup account keys is a pragmatic approach to achive this. To not install it on this AMI and to run it on a separate instance and then copy only the cert files to the AMI will be a cleaner process. To reduce the complexity we can live with the first (pragmatic) approach.

In case of letsencrypt / certbot, the process of creating and renewing should be done (scheduled) as part of the build pipeline for the AMI (amazon machine image), incl. nginx setup.

a) Cert is not installed on a load balancer. Cert is installed on the webserver (nginx).
b) Buildserver is Jenkins CI.
c) Provisioning is done by packer and bash scripting (for this project; usually we use chef)
d) Initial Letsencrypt account registration (key) and inital cert creation could be done manually if this helps to setup the build pipeline or reduces the build pipeline complexity.
e) We want to use the DNS challenge
f) We expect to have less then 100 dns zones on amazon. We expect to have less then 100 domains / subdomains in this one cert on that webserver.
g) We use OCSP stappling.
h) Secrets / credentials / key files should be build plan variables and injected into the build scripts.

DNS challenge is implemented and working:
sudo certbot certonly --staging --manual --expand
–cert-name mycertname
–no-eff-email --agree-tos --manual-public-ip-logging-ok
–manual-auth-hook /tmp/packer/certbot/
–manual-cleanup-hook /tmp/packer/certbot/
(${listOfSwitches} contains “-d” switches, one per subdomain)

I am not sure if the “–manual” plugin is the way we should go with…

Without a full knowledge about all ins and outs of certbot:

  1. What certbot commands would be the ideal sequence to have a build pipeline that can be executed multiple times without user interventions?
  2. What keys/credentials/files should i inject?
  3. Is there a way to re-download the latest issued cert file from letsencrypt, again and again, as a starting point for each build run? Or do i need to pass the cert around from build run to build run? (To enable certbot to say: “no need to renew, cert is not on due”; to shorten build time) Or is there another option?

Hi @c.weiss,

After running it once successfully, I would suggest changing --expand to --keep. Then, if it finds an existing certificate, it will only try to renew it if it’s less than 30 days away from expiring. Otherwise, it will “keep” the existing certificate.

Do you think you could make a complete backup of /etc/letsencrypt, including symbolic links? The symbolic link structure is used for renewal. (At least, you need everything in /etc/letsencrypt/accounts, /etc/letsencrypt/renewal, /etc/letsencrypt/live, and /etc/letsencrypt/archive, including the symbolic links between live and archive.)

This won’t work because you’ll also need the associated private key, which nobody else possesses. (Right now Certbot always generates a new private key for each renewal.)

I don’t think Certbot’s design is super-convenient for what you want to do because we have definitely not optimized for the idea of people (effectively) deleting and recreating /etc/letsencrypt. You might also want to look into some of the other clients like You would probably need to do a bit more scripting, but there might be fewer and simpler filesystem (and package) dependencies.

hi @c.weiss

You do not need change the configuration HOWEVER you will update the file contents

What I mean by that

all your server configuration should point to a specific certificate location HOWEVER the content of that location will need to be updated every 90 days.

There is also an option to use a Networked File System for your certificates.

What I would recommend - a dedicated certificate server (doesn’t have to be big) that will complete challenges and issue new certificates and use a Configuration Management system (Salt, Ansible, Puppet, etc, etc) to disribute the certificates to all your servers.

Once again your configuration (on web servers) will not change but the content of the files they are pointing to will change.


@schoen, thanks.

  1. I expect to get changes to the nginx config when project moves on and i do not want to change my buildscript later on to catch up with changes. Therefore i think i need --expand.

2/3) I will inject the complete dir structure. I think i need to save the dir after each successful build to get a vaild starting point for next build run (artifact). Instead of passing around the latest artifact it would be great to have a never changing dir structure as starting point and to load the lastet issued cert from servers using the existing key (the same key as used for first cert issueing). Is there an command line option for that?

Would be nice if certbot could be extended to allow both fashions of automating the cert process. Splitting the community by automating-fashions would not help the community. I do not like to have many clients, reinventing the wheel and in fact dublicates large parts of the process - just because someone likes go or ruby more then c# or java.

@ahaw021 thanks for sharing a valid option with us.

I do not think we should introduce a new runtime dependency to a network attached storage (e.g. S3 bucket, another “Fileserver”, etc.) because:
a) this “Fileserver” needs to fullfil the high availability (HA) requirements, too
b) this “Fileserver” needs to scale (performance) when the webserver farm scales (AutoScaling)
c) adds network latency or requires caching
d) caching, network and HA adds complexity

Another option is to compose the final server with 3 components: instead of just deploying the “app” on the “server”, we could start deploying “app” and “cert” as separate components to the server, each with an own build and deployment pipeline. Instead of introducing a new “component” to the build and deployment pipeline we treat “cert” as integral part of “server”. Building an cert separately could not speedup cycle time so, we stay with a single build pipeline for the “server”, and keep the build- and deployment complexity / management overhead low.

I would tend to balance all these points to ship the cert with the virtual server image for a few simple reasons:
a) traidoff is we need to deploy new servers more often (at least all 2 month) - is is ok for us - following the continuous delivery credo: if it hurts, do it more often! (to get a less painful process in the long term)
b) having a build-time dependency is alot better then having a runtime dependency (e.g. shorter feedback loop, less runtime problems, etc.)
c) lower complexity

I do not want to introduce a religious war or philosophic dispute, your option is valid and i just balance risk and changes different.

Is it possible that you could also have names removed from your cert? Certbot doesn’t handle that case well at all unless you specify --cert-name.

I’m working on a feature to reuse keys, which should be available in a new release soon. But it currently doesn’t have a concept of downloading existing keys. I agree that might be a useful feature and I’m happy to consider it.

Let’s Encrypt itself doesn’t have a feature to download existing certificates, but they are currently available from (which is operated by Comodo). To download the certificates, Certbot would probably have to use that service or some other interface to Certificate Transparency logs.

1 Like

@schoen thanks. About removed subdomains: is i expect that a subdomain could be removed when project moves on. Even the main domain could change. This is why i added --cert-name to my first post. I use a name e.g. “projectfoo” that is not related to any domain/subdomain, to prevent certbot from picking a domain-based cert name. For me it is not absolutely clear if certbot cleanup (remove) no longer existing subdomains from cert if i access the cert with --cert-name and --expand and while adding all subdomains as -d options and omitting no longer existing subdomains. If no longer existing subdomains remains in cert this is not clean but not catastrophic. In a project with nearly 100 subdomains or with many changing subdomains it could become a problem, because a letsencrypt cert is limited to 100 alternative names - and “dead subdomains” that remains in a renewed cert could eat up space we need for other subdomains.

+1 for reusing keys

Would be nice if there will be a concept for re-downloading already issues certs.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.