Manual creation of /etc/letsencrypt for bootstrapping servers with pre-existing cert

matp · June 26, 2021, 8:27am

My domain is: not relevant

I ran this command: using certonly for renewals but bootstrapping server with existing certificate because during initial deployments we test the server before making DNS change...so letsencrypt can't perform domain verification against new server. So we set everything up with a valid cert and then within X days of expiration (checked with openssl) we tell certbot to do a certonly so we get a new cert.

It produced this output: live directory exists for...
i.e. doesn't overwrite/replace default/bootstrap certificate and instead places the new cert in an alternative (non-standard) directory (numerical suffix).
Our provisioning sees the cert at the default location is still expiring within X days and so keeps attempting to generate a letsencrypt cert and of course runs into rate limits.

My web server is (include version): not relevant

The operating system my web server runs on is (include version): not relevant

My hosting provider, if applicable, is: not relevant

This caught my eye...

There is a way to create the entire /etc/letsencrypt structure with a dummy cert from the outset, but it’s not documented and requires a bit of manual work, so I don’t think I would suggest that.

Our provisioning/orchestration manages nginx configs and we don't want other utilities that we can't easily monitor making changes as this will result in the config flapping back and forth between what letsencrypt wants and what our provisioning wants.

So we use certonly. When we can set DNS before deploying server then everything is fine because letsencrypt can generate the first certificate and successfully does domain verification.

However when moving a server we often want to test production configs before making the DNS change. This often means deploying the new server and setting record in hosts file to verify everything (including https) is working. Many apps depend on having base urls configured (https://f.q.d.n/p/a/t/h) to function properly so we do this hosts file based test....but https needs to work and letsencrypt is not an option without domain verification.

So we deploy a bootstrap certificate that is valid for that domain in the letsencrypt location.
Our provisioning/orchestration periodically checks that cert for expiration (using openssl) and when it is within X days of expiration, asks letsencrypt to generate a new cert (ie the bootstrap cert gets replaced with new letsencrypt cert). So we really only run certbot once every 60 days or so. The bootstrap cert might be letsencrypt transferred from an existing production box or might be from a different CA.

We could put this bootstrap cert in a different directory and then manage switching around nginx configs (pointing at different cert paths) when we switch...but that makes the process a lot more complicated of course.

So....long story short.... wondering if there is some trick to how we are constructing our certbot directory with bootstrap cert so that when we run certbot it will overwrite the cert that is there (the bootstrap cert) without complaining live directory exists for example.com and then places the cert in a non-standard directory.

What appears to be happening is that the cert is saved but to a non-standard path.

Debug Log

2021-06-26 00:13:01,022:INFO:certbot._internal.client:Non-standard path(s), might not work with crontab installed by your operating system package manager
2021-06-26 00:13:01,023:DEBUG:certbot._internal.storage:Creating directory /opt/certbot/archive/example.com-0001.
2021-06-26 00:13:01,023:DEBUG:certbot._internal.storage:Creating directory /opt/certbot/live/example.com-0001.
2021-06-26 00:13:01,023:DEBUG:certbot._internal.storage:Writing certificate to /opt/certbot/live/example.com-0001/cert.pem.
2021-06-26 00:13:01,023:DEBUG:certbot._internal.storage:Writing private key to /opt/certbot/live/example.com-0001/privkey.pem.
2021-06-26 00:13:01,023:DEBUG:certbot._internal.storage:Writing chain to /opt/certbot/live/example.com-0001/chain.pem.
2021-06-26 00:13:01,024:DEBUG:certbot._internal.storage:Writing full chain to /opt/certbot/live/example.com-0001/fullchain.pem.
2021-06-26 00:13:01,024:DEBUG:certbot._internal.storage:Writing README to /opt/certbot/live/example.com-0001/README.
2021-06-26 00:13:01,061:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer <certbot._internal.cli.cli_utils._Default object at 0x7fb8ba5cd910>
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var server=https://acme-v02.api.letsencrypt.org/directory (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var account={'server'} (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var rsa_key_size=2048 (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var key_type=rsa (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var config_dir=/opt/certbot (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var server=https://acme-v02.api.letsencrypt.org/directory (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var authenticator=webroot (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var webroot_path=/var/www/example.com (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var webroot_path=/var/www/example.com (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2021-06-26 00:13:01,061:DEBUG:certbot._internal.storage:Writing new config /opt/certbot/renewal/example.com-0001.conf.
2021-06-26 00:13:01,063:DEBUG:certbot.display.util:Notifying user:
Successfully received certificate.
Certificate is saved at: /opt/certbot/live/example.com-0001/fullchain.pem
Key is saved at:         /opt/certbot/live/example.com-0001/privkey.pem
This certificate expires on 2021-09-23.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

You can see we expect the cert to be located in /opt/certbot/live/example.com but because of the existence of the bootstrap cert in that location, certbot places the new cert at /opt/certbot/live/example.com-0001 and subsequent attempts (because we still see the cert at /opt/certbot/live/example.com is still expiring soon) end up at -0002, -0003, etc until we run into the rate limit.

root@some-server:~# ls -l /opt/certbot/live/
total 28
drwxr-xr-x 2 root root 4096 Apr 29 09:55 example.com
drwxr-xr-x 2 root root 4096 Jun 26 00:13 example.com-0001
drwxr-xr-x 2 root root 4096 Jun 26 00:18 example.com-0002
drwxr-xr-x 2 root root 4096 Jun 26 00:23 example.com-0003
drwxr-xr-x 2 root root 4096 Jun 26 00:28 example.com-0004

I suppose another way to "hack" this bootstrap is to check for the presence of the -0001 directory after requesting a cert and if it is there, replace the correct location with the -0001 directory.

_az · June 26, 2021, 9:00pm

To put it simply; it's not possible to manually bootstrap an external certificate like this.

Certbot is quite sensitive about the way it stores things. If something is missing (like in your case, the /etc/letsencrypt/renewal/example.com.conf file), then it gets confused and things will go wrong.

I can think of two ways you can go about this:

If you're always bootstrapping a certificate which was obtained with Certbot on another server, then tar up /etc/letsencrypt/ and extract it on the destination server. This will work because all the relevant files and symlinks will be preserved.
Otherwise, If you want to be able to boostrap an external certificate, then just use an additional symlink to allow a clean transition from one state to the next. e.g.
1. Have /etc/ssl/crt.key and /etc/ssl/crt.pem initially symlinked to your non-Certbot certificate.
2. When Certbot runs, include a --deploy-hook or something which will re-link those files to/etc/letsencrypt/live/example.com/privkey.pem and /etc/letsencrypt/live/example.com/fullchain.pem, respectively.

rg305 · June 27, 2021, 12:13am

Enter the "IF" statement...

IF a certbot cert exists
    then use that one
ELSE
    then use the bootstrapped cert

[I leave you to workout the adequate syntax for your software]

matp · June 27, 2021, 7:22am

Hey thanks for the detailed answer. The --deploy-hook looks like it might be quite useful to solve this cleanly.

Currently there are two independent provisioning steps:

if cert in /path/to/letsencrypt/live/example.com/ does not exist or is expiring within X days (checked with openssl), then run certbot to try to obtain a certificate.
- So if DNS isn't in place, that won't work (certbot will fail) and we manually drop the bootstrap cert in the expected spot. Then our provisioning will stop trying to run certbot (because valid cert in place according to openssl) and our provisioning and testing can successfully complete.
if the cert/key in /path/to/letsencrypt/live/example.com/ is different than the one in our webserver certs directory, then copy (if-diff-then-copy).

The reason we are doing these individually and not letting certbot "do too much" on its own is that we track, monitor and report every change made to to the servers over time. That means external scripts/programs need to do one thing at a time (or not) and we generally test whether we need to do work before we actually run the command so that the reporting is cleaner (if we run certbot, we assume a change was made...so we test whether we think we need to run certbot before we run certbot). Same reason we use if-diff-then-copy rather than just using rsync and letting rsync decide whether a copy is needed or not. This allows us to see flapping/conflicting config file changes/edits or if someone has manually changed something on the system because all of a sudden we will see changes were necessary to correct something.

Anyway. Probably too much detail. But I appreciate the advice you shared with me as it will help me come up with a reasonable solution without a bunch of additional conditionals to describe this edge case.

What I will probably do...

change our cert-file-exists and openssl expiration tests to check against a "deploy path" rather than letsencrypt's dir path.
run certbot with --deploy-hook to copy certs to a "deploy path"
- If certbot fails (e.g. DNS not yet setup) and our provisioning is failing, then I can manually drop a valid cert at the "deploy path" and our provisioning will stop trying to run certbot.
Later on when server is up and running (DNS in place) and the bootstrap cert is expiring, running certbot with the --deploy-hook will replace the bootstrap cert with a letsencrypt cert.

matp · June 27, 2021, 8:09am

I appreciate you taking the time to reply. I definitely sometimes miss the obvious (forest, trees and all that).

In this case the problem is that there is not really a bootstrap certificate on the server (wasn't precise about the process in my post, sorry). After starting the new server up (provisioning will probably fail due to certbot failure before we get a chance to ssh in the first time). certbot will keep failing because DNS is not in place and might not be for a few days (e.g. because we aren't going to migrate the domain name until we have finished testing the new app). So we manually drop a valid cert on the server and the provisioning stops trying to run certbot and we can add an entry to our hosts file and test the new server/app.

We try to avoid doing manual config management on any servers, but with some cases it is unavoidable (for us) as the time/cost of automating certain edge cases is not worth it (additional certificate/credential management complexity, etc).

If we do end up using the bootstrap cert because DNS isn't yet setup....I want to go ahead and keep using that cert up until expiration...instead of trying to run certbot constantly (and failing until DNS is updated)....this makes using conditional logic more complex.

I could certainly add this in as a standard case, define a separate standard path to look for bootstrap cert, and then setup all the conditionals for when to use what...

IF (! certbotCertExists || certBotCertExpiresDays < 5) && (!bootstrapCertDeployed || bootstrapCertExpiresDays < 5)
    IF runCertBot
        // deploy certbot cert
    ELSE IF bootstrapCertExists && bootstrapCertExpiresDays > 4
        // deploy bootstrap cert
    ELSE
        // fail
ELSE
    // nothing to do

But it is much cleaner to just say:

IF the cert at the "standard path" exists and doesn't expire for at least X days
    // do nothing.  
ELSE
    // try to generate a new cert with certbot and maybe fail.

// if we are failing, someone manually can place a valid cert at the "standard path" to resolve the failure

The problem here is that I was using the letsencrypt dir as the "standard path" and certbot doesn't like that.

...I think with a small change (using the --deploy-hook to copy the certbot cert out of the letsencrypt dir structure) we can easily stick with this simplified logic (just using different "deploy dir" as "standard path" rather than the letsencrypt dir).

rg305 · June 27, 2021, 9:54pm

Then don't use that path at all (nothing is forcing you to do so).
Use any other path for the non-certbot generated certs.
The IF statement can likely also be used within the HTTPS vhost config.
If so, then the choices within that block are more clearly defined and separated.
[You can always call certbot twice a day - even before any cert has been issued]

system · July 27, 2021, 9:55pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.