Certbot does not respect `cert-name` when it comes to path naming on disk

Help
1

Update: I solved my own issue but leaving this here in case anyone else runs into the same thing (as I already typed it up). TL;DR: if you move your certbot config_dir you need to manually update the paths in the renewal/*.conf files.

Switched from snap to pip install of certbot. Presumably certbot version was updated at the time of change (I didn't note the pre-change certbot version). Now running certbot 5.6.0 via pip.

We use certbot to issue/renew certs (we manage that process externally and don't use the built-in renewal daemon because we need to know when the cert renews so we can restart/reload services that use the certs). We also don't use certbot to install any certs. We create our own symlinks outside of the certbot data dir pointing at the current cert. So we rely on predictable certificate paths on disk.

e.g. for --cert-name DOMAIN_NAME I expect to find the cert at <DATA_DIR>/live/DOMAIN_NAME/cert.pem. Since the certbot update the certificates are being generated in locations with index numbers like <DATA_DIR>/live/DOMAIN_NAME-XXXX/cert.pem. Of course this means our symlinks don't know where to point and we also can't easily check expiration because we don't know where to look for the files.

certbot certonly --cert-name 
certbot certonly --webroot --webroot-path /var/www/DOMAIN_NAME --domains DOMAIN_NAME --non-interactive --key-type rsa --rsa-key-size 2048 --cert-name DOMAIN_NAME --config-dir /opt/certbot/data --server https://acme-v02.api.letsencrypt.org/directory --agree-tos --email it@DOMAIN_NAME


Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for DOMAIN_NAME
Successfully received certificate.
Certificate is saved at: /opt/certbot/data/live/DOMAIN_NAME-0005/fullchain.pem
Key is saved at:         /opt/certbot/data/live/DOMAIN_NAME-0005/privkey.pem
This certificate expires on 2026-09-30.
These files will be updated when the certificate renews.

Running certbot certificates reveals some possible underlying issues...

The following renewal configurations were invalid:
  /opt/certbot/data/renewal/DOMAIN_NAME.conf

and

Renewal configuration file /opt/certbot/data/renewal/DOMAIN_NAME.conf produced an unexpected error: expected /opt/certbot/live/DOMAIN_NAME/cert.pem to be a symlink.

Maybe this is artifact of a certbot behaviour change and the right answer is just to blow away the data dir and start from scratch? The symlink bit is weird because checking the certbot data dir, none of the new certs are symlinks either.

Whatever the fix is, I need to apply this automatically across a lot of servers. Due to the nature of the problem and our setup, the automation has also run into a problem: because the certificate on the expected path is expiring soon and so we try to renew over and over which either creates new certs in unexpected paths or fails due to too many requests.

So I would prefer to fix the issue in place. e.g. fix any issues with the config and folder structure and delete unnecessary files/certs. If I blow away the certbot data directory, then we will have an SSL outage until certbot manages to retrieve a new certificate.

certbot certificates 
# certbot certificates --config-dir /opt/certbot/data

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /opt/certbot/data/renewal/DOMAIN_NAME.conf produced an unexpected error: expected /opt/certbot/live/DOMAIN_NAME/cert.pem to be a symlink. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: DOMAIN_NAME-0001
    Serial Number: 5158013d0adf25251ebff3b43286ab67778
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-28 14:41:43+00:00 (VALID: 84 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0001/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0001/privkey.pem
  Certificate Name: DOMAIN_NAME-0002
    Serial Number: 51c54c39132f0f3b2948ea9891b8463d538
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-28 14:56:36+00:00 (VALID: 84 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0002/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0002/privkey.pem
  Certificate Name: DOMAIN_NAME-0003
    Serial Number: 581a615a22214cadcba387faa2aec5eba23
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-28 15:06:41+00:00 (VALID: 84 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0003/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0003/privkey.pem
  Certificate Name: DOMAIN_NAME-0004
    Serial Number: 5f3ef2554f5f67c23ae56825288cc843e39
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-28 15:16:38+00:00 (VALID: 84 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0004/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0004/privkey.pem
  Certificate Name: DOMAIN_NAME-0005
    Serial Number: 57fbfe4c9bccf020130b95e270cff1ae72e
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 00:21:59+00:00 (VALID: 85 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0005/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0005/privkey.pem
  Certificate Name: DOMAIN_NAME-0006
    Serial Number: 6fe839960e9a72d9d4817c2471e7b294a5a
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 18:22:07+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0006/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0006/privkey.pem
  Certificate Name: DOMAIN_NAME-0007
    Serial Number: 63bde861c7c2460bc3da09b13809ec6c7a9
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 18:27:07+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0007/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0007/privkey.pem
  Certificate Name: DOMAIN_NAME-0008
    Serial Number: 6c6ad8bbd755378dc63839f48be8d168ab0
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 18:32:09+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0008/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0008/privkey.pem
  Certificate Name: DOMAIN_NAME-0009
    Serial Number: 6f4ea4bca6e684747ee43be5bd63046c70d
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 18:37:15+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0009/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0009/privkey.pem
  Certificate Name: DOMAIN_NAME-0010
    Serial Number: 6c2317b78785b505929c6d53fed26e987e5
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 18:42:06+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0010/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0010/privkey.pem
  Certificate Name: DOMAIN_NAME-0011
    Serial Number: 583688b82056dd82450ae307abb1d9c77df
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 21:52:11+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0011/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0011/privkey.pem
  Certificate Name: DOMAIN_NAME-0012
    Serial Number: 50e480de9dceaf1a73759708d4a21cfa8b1
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 21:57:11+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0012/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0012/privkey.pem
  Certificate Name: DOMAIN_NAME-0013
    Serial Number: 52ab38e9ab21239312692ff9435e7a0dd9a
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 22:07:10+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0013/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0013/privkey.pem
  Certificate Name: DOMAIN_NAME-0014
    Serial Number: 5691360e8ad99c0b4cb40935cda70b0639d
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-09-30 22:17:11+00:00 (VALID: 86 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0014/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0014/privkey.pem
  Certificate Name: DOMAIN_NAME-0015
    Serial Number: 57e3fea520e48ac9ae22f775f4c5bd674d0
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-10-02 07:32:28+00:00 (VALID: 87 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0015/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0015/privkey.pem
  Certificate Name: DOMAIN_NAME-0016
    Serial Number: 506e9299b08f97baaf0c4650c5f787c4f0b
    Key Type: RSA
    Identifiers: DOMAIN_NAME
    Expiry Date: 2026-10-03 17:07:47+00:00 (VALID: 89 days)
    Certificate Path: /opt/certbot/data/live/DOMAIN_NAME-0016/fullchain.pem
    Private Key Path: /opt/certbot/data/live/DOMAIN_NAME-0016/privkey.pem

The following renewal configurations were invalid:
  /opt/certbot/data/renewal/DOMAIN_NAME.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Files in certbot data dir 
# find /opt/certbot/data/ \( -type f -o -type l \) -exec ls -la {} \;
/opt/certbot/data/renewal/DOMAIN_NAME.conf
/opt/certbot/data/accounts/acme-v02.api.letsencrypt.org/directory/9bbbb15ffe1874814cd0810abc9449cd/private_key.json
/opt/certbot/data/accounts/acme-v02.api.letsencrypt.org/directory/9bbbb15ffe1874814cd0810abc9449cd/meta.json
/opt/certbot/data/accounts/acme-v02.api.letsencrypt.org/directory/9bbbb15ffe1874814cd0810abc9449cd/regr.json
/opt/certbot/data/keys/0000_key-certbot.pem
/opt/certbot/data/keys/0001_key-certbot.pem
/opt/certbot/data/keys/0003_key-certbot.pem
/opt/certbot/data/keys/0002_key-certbot.pem
/opt/certbot/data/csr/0002_csr-certbot.pem
/opt/certbot/data/csr/0003_csr-certbot.pem
/opt/certbot/data/csr/0000_csr-certbot.pem
/opt/certbot/data/csr/0001_csr-certbot.pem
/opt/certbot/data/live/README
/opt/certbot/data/live/DOMAIN_NAME/README
/opt/certbot/data/live/DOMAIN_NAME/privkey.pem -> ../../archive/DOMAIN_NAME/privkey20.pem
/opt/certbot/data/live/DOMAIN_NAME/chain.pem -> ../../archive/DOMAIN_NAME/chain20.pem
/opt/certbot/data/live/DOMAIN_NAME/cert.pem -> ../../archive/DOMAIN_NAME/cert20.pem
/opt/certbot/data/live/DOMAIN_NAME/fullchain.pem -> ../../archive/DOMAIN_NAME/fullchain20.pem
/opt/certbot/data/archive/DOMAIN_NAME/fullchain16.pem
/opt/certbot/data/archive/DOMAIN_NAME/cert15.pem
/opt/certbot/data/archive/DOMAIN_NAME/chain17.pem
/opt/certbot/data/archive/DOMAIN_NAME/cert20.pem
/opt/certbot/data/archive/DOMAIN_NAME/privkey17.pem
/opt/certbot/data/archive/DOMAIN_NAME/privkey19.pem
/opt/certbot/data/archive/DOMAIN_NAME/chain18.pem
/opt/certbot/data/archive/DOMAIN_NAME/cert18.pem
/opt/certbot/data/archive/DOMAIN_NAME/privkey15.pem
/opt/certbot/data/archive/DOMAIN_NAME/privkey20.pem
/opt/certbot/data/archive/DOMAIN_NAME/chain15.pem
/opt/certbot/data/archive/DOMAIN_NAME/fullchain20.pem
/opt/certbot/data/archive/DOMAIN_NAME/fullchain18.pem
/opt/certbot/data/archive/DOMAIN_NAME/privkey16.pem
/opt/certbot/data/archive/DOMAIN_NAME/chain19.pem
/opt/certbot/data/archive/DOMAIN_NAME/fullchain17.pem
/opt/certbot/data/archive/DOMAIN_NAME/cert19.pem
/opt/certbot/data/archive/DOMAIN_NAME/privkey18.pem
/opt/certbot/data/archive/DOMAIN_NAME/cert16.pem
/opt/certbot/data/archive/DOMAIN_NAME/cert17.pem
/opt/certbot/data/archive/DOMAIN_NAME/chain16.pem
/opt/certbot/data/archive/DOMAIN_NAME/fullchain15.pem
/opt/certbot/data/archive/DOMAIN_NAME/chain20.pem
/opt/certbot/data/archive/DOMAIN_NAME/fullchain19.pem

solution

Posting this in case anyone else runs into the same problem.

I had moved the certbot data dir as part of the new setup when switching from snap to pip. Forgot about that "variable" when considering potential causes.

The .conf files in the DATA_DIR/renewal/ directory contain hard-coded absolute paths. When running certbot certonly or certbot certificates with the new DATA_DIR, it finds these config files and then looks on the hard-coded paths for the live (symlinks) and archive dirs. As the paths in the conf file are out of date, it does not find these and complains about the missing symlinks. So it sees the config file as invalid and needs to use a new cert-name to avoid collisions (either to avoid clobbering the renewal/CERT_NAME.conf file or the existing {live,archive}/CERT_NAME/ dir). So it adds the numbering on the end of the certname. Each time our own external renewal process runs with the original cert name the process happens again including collisions with the various numbering.

The solution is the fix the paths in the renewal/*.conf files and purge all the extra files with index numbers in renewal/ and live/ and archive/ dirs.

While this is a user-caused edge case, it would be nice if certbot logged when this was happening to make the root cause more obvious. e.g. Collision with cert-name _____ due to existing invalid config renewal/CERT_NAME.conf. Will attempt to use cert-name _____ instead.. Presumably the renewal live and archive dirs will always live in the same DATA_DIR/config_dir... so could also add support for using a placeholder/variable for $config_dir in the renewal .conf files to make it easier to move the directory (i.e. paths relative to config_dir rather than absolute paths).

Just an idea.

2

Just so you know, the standard way to deal with that is to pass a --deploy-hook to certbot of a script that runs whatever tasks you need completed when a new certificate is available. Having certbot manage the renewal process means that you don't need to worry about when a CA issue means the certificate needs to renew earlier than originally scheduled, or when certificate lifetimes change, or whatever. Though certainly one can manage that outside of certbot if one wants to, but it can be more complicated than just running the script every couple months.

And again, while that approach can certainly work, the standard approach would be to use the deploy hook to copy the certbot-managed files to wherever else on your system you want them, with the permissions/path/etc. that you actually need.

While there may be people from the certbot team that read here occasionally, probably the best place to post a feature request for them is on their Github issues.

3

Appreciate the feedback on the certbot intended way of doing things. Always good to have that for context and to look if there is a better way of doing things for a given situation.

A bit more context on our setup/use-case and motivations for how we do things. I won't be offended if you ignore this. :slight_smile:

We have separate process that is part of our devops stuff that checks and decides when a certificate needs to be created or renewed which then invokes certbot. Certbot does not have cron/timer for automated renewals, so the control and monitoring is in the hands of devops system directly.

This is generic and so may be used for many things.

Using deploy-hook means we need to generate and have devops manage custom hook scripts for every possible use case and, path, server config permutation, etc. But does have the advantage that certbot sets $RENEWED_LINEAGE so you can find the cert on disk. But turns out finding the cert on disk is not a problem when everything is working via config_dir and cert-name.

We also have the challenge that we need the devops environment to manage renewal configs and crons or systemd timers for putting certbot in charge of automating renewals. For instance removing renewal configs for certs that are no longer used on the box ensuring the systemd timer for certbot is running and not failing.

Our monitoring uses openssl to test necessary certs for validity and nearing expiration, this includes non-public certs that can't be easily measured on a public http endpoint. This monitoring will still need to know where the current certs are on disk in order to provide artifact monitoring in case certbot has a problem...not to mention adding some type of monitoring for the certbot automated process itself.

For our situation, inverting control to put certbot in the driver's seat would require quite a lot of changes, would be more work to maintain and add new apps and doesn't add (enough) value over the setup we have today to make it worth it. I understand the point about perhaps early renewals due to CA issues rather than just expiration. A simpler way for us to handle that given our setup would be to just add an additional periodic dry-run check to certbot to detect this additional edge-case when a cert should be renewed even if not within our own expiration/renewal window.

Our devops environment tracks actual changes made as a metric (vs testing conditions/config are in the desired state). This allows us to identify when devops config might be conflicting and oscillating back and forth between two states or when something has happened that results in a large change on the machine. Yes, certs could be considered something that should be an independent pillar that works in tandem with the devops environment, and indeed this is the direction we are moving in...(our own PKI as well as letsencrypt signing moving away from individual servers).

4

Oh, I certainly understand that once you have a working system there's a lot of hesitation to change it. I think I wouldn't have chosen certbot specifically as the ACME client implementation if I were going to be handling everything outside of it like that, but I suppose it works adequately.

I don't think calling certbot with a dry-run will detect whether the certificates need to be renewed at an unusual time. If you're going to take full ownership of the renewal schedule yourself, I think you're going to want to implement checking ARI yourself to use it as part of your scheduling algorithm.