Support import of existing certificate(s) with certbot

grenade · May 31, 2022, 8:44am

question:

how do we make certbot aware of the existence of certs which have been provisioned by mechanisms other than running certbot to obtain the cert?

here is our use case:

we never use wildcard certificates.

we provision new cloud instances on a very regular basis using terraform and other automation technologies.

sometimes an instance has issues that occur after certbot has successfully requested and received a certificate against its fqdn. often the simplest solution is to destroy the problematic instance and re-provision a new instance with the same fqdn. when we do this, our automation securely stores a copy of the certbot certificate (all four archive pem files) in our secret store and deploys that copy (and the live symlinks) to the re-provisioned instance using the expected filesystem permissions and ownerships. we learned the hard way that if we don't do that, we get throttled by letsencrypt for requesting new certificates for fqdns that we have already been issued certificates against (ie: when an instance deployment has had multiple consecutive deployment failures). if that happens we must wait a week to request a new cert for the fqdn. this is why we always keep a copy of issued certs in our secret store. it's also useful if we need to move a deployed instance to a different geography or cloud provider.

problems occur when certificates that have been deployed from our secret store expire. this is because the certbot service on these instances was never aware of the certificate existence. our re-provisioning process places certs in the conventional locations (/etc/letsencrypt/{live,archive}) but we've not yet figured out how to make certbot aware that it should renew these certs on expiry.

actually, writing the above has made me realise that it's quite likely that if we were to also populate /etc/letsencrypt/renewal with a correctly structured .conf file, the certbot service would include the re-provisioned cert in its renewal processes. i may have answered my own question.

testing creating the missing renewal .conf theory now...

ok, that worked.

a possible answer:

here's what i did (i already had authentic, but soon to expire letsencrypt certs in the archive folder and symlinks in the live folder, i assume this is significant because it allows the certbot client to authenticate properly):

state at start:

$ sudo ls -ahl /etc/letsencrypt/{live,archive,renewal}
/etc/letsencrypt/archive:
total 12K
drwx------ 5 root root 4.0K Jan 20 11:07 .
drwxr-xr-x 9 root root 4.0K May 31 07:27 ..
drwxr-xr-x 2 root root 4.0K Mar 18 13:51 pectinata.manta.systems

/etc/letsencrypt/live:
total 16K
drwx------ 5 root root 4.0K Jan 20 11:07 .
drwxr-xr-x 9 root root 4.0K May 31 07:27 ..
-rw-r--r-- 1 root root  740 Nov 12  2021 README
drwxr-xr-x 2 root root 4.0K Mar 18 13:51 pectinata.manta.systems

/etc/letsencrypt/renewal:
total 8.0K
drwxr-xr-x 2 root root 4.0K May 30 13:39 .
drwxr-xr-x 9 root root 4.0K May 31 07:27 ..

$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certs found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

create the renewal .conf:

cert_name=$(hostname -f)
client_version=$(certbot --version | cut -d ' ' -f2)
client_account=$(sudo ls /etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory)

sudo bash -c "cat << EOF > /etc/letsencrypt/renewal/${cert_name}.conf
# renew_before_expiry = 30 days
version = ${client_version}
archive_dir = /etc/letsencrypt/archive/${cert_name}
cert = /etc/letsencrypt/live/${cert_name}/cert.pem
privkey = /etc/letsencrypt/live/${cert_name}/privkey.pem
chain = /etc/letsencrypt/live/${cert_name}/chain.pem
fullchain = /etc/letsencrypt/live/${cert_name}/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = ${client_account}
pref_challs = http-01,
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
EOF"

check if certbot is now aware of the cert:

$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: pectinata.manta.systems
    Domains: pectinata.manta.systems para.metrics.pectinata.manta.systems relay.metrics.pectinata.manta.systems rpc.pectinata.manta.systems
    Expiry Date: 2022-06-16 12:51:13+00:00 (VALID: 16 days)
    Certificate Path: /etc/letsencrypt/live/pectinata.manta.systems/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pectinata.manta.systems/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

attempt renewal:

$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/pectinata.manta.systems.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for para.metrics.pectinata.manta.systems
http-01 challenge for pectinata.manta.systems
http-01 challenge for relay.metrics.pectinata.manta.systems
http-01 challenge for rpc.pectinata.manta.systems
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/pectinata.manta.systems/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/pectinata.manta.systems/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

state at completion:

$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: pectinata.manta.systems
    Domains: pectinata.manta.systems para.metrics.pectinata.manta.systems relay.metrics.pectinata.manta.systems rpc.pectinata.manta.systems
    Expiry Date: 2022-08-29 07:11:11+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/pectinata.manta.systems/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pectinata.manta.systems/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

great. our problem is solved, but back to my feature request. all the information required for this import was present on the system (as demonstrated above). wouldn't it be great if i could have run a certbot command to do all this? eg:

# when certs are in conventional location
sudo certbot import --cert-name $(hostname -f) [ --standalone, etc... ]
# when certs are elsewhere
sudo certbot import --path /tmp/$(hostname -f) [ --standalone, etc... ]

Osiris · May 31, 2022, 9:20am

while I'm not a Certbot engineer, I'm not sure if this is wise. It's not recommended to manually mess with the contents of the /etc/letsencrypt/ directory in general. If a user wants to do something with that directory, usually we recommend to backup or sync it entirely, preserving symbolic links et cetera. Not just parts of it.

Also, it seems your feature request comes from the fact your previous backup strategy was:

Pull the files from /etc/letsencrypt/{live,archive}/${certificatename}

Whereas you, as you already figured out, should have done:

Pull files from /etc/letsencrypt/{live,archive}/${certificatename} and /etc/letsencrypt/renewal/${certname}.conf.

This is not a large addition to any backup strategy, so I'm not sure if it warrants the implementation of an entire new function which would probably cost a lot of development time. Also taking into account even simple additions/bugfixes take enormous amounts of time due to the fact the Certbot development team is very sparsely populated unfortunately.

grenade · May 31, 2022, 9:40am

of course backing up the renewal file would work in some scenarios but it's a little brittle in that it makes assumptions about authenticator (preferred challenges) and installer (standalone, etc) that may not be valid in a redeployment scenario.
i can imagine many scenarios (search import certificate in this forum) where certificate imports are legitimate use cases different to our own.
i appreciate that development time is valuable and limited and that this is a non-trivial feature request but i don't see why we shouldn't have the conversation. our own use case is resolved by today's discovery about renewal config and i'm comfortable adding generation of these configs to our automation but i think certbot would be missing a trick to dismiss this feature request especially considering the stance on discouraging messing with the contents of /etc/letsencrypt by processes other than certbot.

Osiris · May 31, 2022, 10:16am

But those can be overridden using the appropriate command line options. If I read your examples correctly, you'd have to specify these things with the import command too, as Certbot wouldn't be able to guess it for you. Or the import command would also use the renewal configuration file if no command line options were used. So I don't really see the difference between importing an existing lineage and using the renewal configuration file directly.

grenade · May 31, 2022, 10:19am

you said yourself that messing with the contents of /etc/letsencrypt is discouraged. this feature request, if implemented, would allow for users to use certbot, rather than another process to perform the import. as it stands, there is no mechanism (afaik) other than messing with the contents of /etc/letsencrypt to perform an import.

Osiris · May 31, 2022, 11:00am

That's certainly a good point indeed. Syncing an entire /etc/letsencrypt/ directory misses the granularity to import only certain certificates instead of just everything.

jvanasco · May 31, 2022, 2:45pm

You'll need to request this on the Certbot repository: GitHub - certbot/certbot: Certbot is EFF's tool to obtain certs from Let's Encrypt and (optionally) auto-enable HTTPS on your server. It can also act as a client for any other CA that uses the ACME protocol.

People here can help you with Certbot problems, and there are some Certbot devs here, but they have their own bugtracker for this stuff. Requests here are really supposed to be for ISRG/LetsEncrypt's ACME service these days; there are a lot of legacy links still pointing to here from the days when both projects were a single team. Certbot was split out and taken over by EFF several years ago.

While I support your idea in general, I want to note a few things:

1- Based on what you shared, I don't think Certbot is the right solution to manage your cloud. IMHO, I think you're halfway to the right solution - having your automated systems pull things from a shared secret store. If I were you, I would either use something like Caddy – which can access the store and autopopulate as needed – or have all machines proxy/redirect ACME traffic to a single node that is responsible for obtaining certs and populating the shared storage. We open sourced our cloud system, Peter SSLers but it's out-of-sync with the functional internal tools and I haven't backported things yet.

2- the import function can not work on a $hostname, it should only use the actual cert-name, as reflected in the directory structure for the cert and renewal info. You're never guaranteed a cert-name will be a simple hostname - in a multi-FQDN cert, the cert-name could be the name of any FQDN, and Certbot has some subroutines that can create/utilize versioned directories.

I think a doable command might be something like this...

# import cert and renewal config
sudo certbot import --cert-name $CERT_NAME --alternate-directory /mounts/other-volume/etc/letsencrypt

# only import the cert
sudo certbot import --cert-name $CERT_NAME --cert-directory /path/to/files

system · June 30, 2022, 2:46pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.