How does certbot decide the -000x suffix for certificate renewal

Help
1

I am unable to understand why certbot command reported failures for .com, .com-0001, -0002 and -0004 but issued certificate under directory with prefix -0003
This resulted in a broken cert issuer code on my server side that thinks 0004 must be the latest

How may I avoid this to have a single directory with latest certificate ?

My domain is: egnyte-appliance.com

I ran this command: ['--webroot', '--agree-tos', '--email', 'ss_certs@egnyte.com', '-n', '--config-dir', '/opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com', '--logs-dir', '/opt/certservice/3.3.0.1652419071/LE/logs', '--work-dir', '/opt/certservice/3.3.0.1652419071/LE/work', '-w', '/usr/share/nginx/html/letsencrypt', '-d', 'bjns60jwh9.qa.egnyte-appliance.com', '--force-renewal']

It produced this output:

2023-10-17 14:38:17,845:DEBUG:certbot._internal.cert_manager:Renewal conf file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0001.conf is broken. Skipping.
2023-10-17 14:38:17,845:DEBUG:certbot._internal.cert_manager:Traceback was:
Traceback (most recent call last):
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/cert_manager.py", line 444, in _search_lineages
    candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 498, in __init__
    self._check_symlinks()
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 572, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /opt/certservice/3.3.0.1642516670/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0001/cert.pem to be a symlink

2023-10-17 14:38:17,846:DEBUG:certbot._internal.cert_manager:Renewal conf file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0002.conf is broken. Skipping.
2023-10-17 14:38:17,846:DEBUG:certbot._internal.cert_manager:Traceback was:
Traceback (most recent call last):
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/cert_manager.py", line 444, in _search_lineages
    candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 498, in __init__
    self._check_symlinks()
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 572, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /opt/certservice/3.3.0.1645093818/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0002/cert.pem to be a symlink

2023-10-17 14:38:17,849:DEBUG:certbot._internal.cert_manager:Renewal conf file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0004.conf is broken. Skipping.
2023-10-17 14:38:17,849:DEBUG:certbot._internal.cert_manager:Traceback was:
Traceback (most recent call last):
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/cert_manager.py", line 444, in _search_lineages
    candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 498, in __init__
    self._check_symlinks()
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 572, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /opt/certservice/3.3.0.1649426201/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0004/cert.pem to be a symlink
2023-10-17 14:38:17,850:DEBUG:certbot._internal.cert_manager:Renewal conf file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com.conf is broken. Skipping.
2023-10-17 14:38:17,850:DEBUG:certbot._internal.cert_manager:Traceback was:
Traceback (most recent call last):
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/cert_manager.py", line 444, in _search_lineages
    candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 498, in __init__
    self._check_symlinks()
  File "/opt/certservice/3.3.0.1652419071/venv/lib/python3.6/site-packages/certbot/_internal/storage.py", line 572, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /opt/certservice/3.3.0.1641979321/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com/cert.pem to be a symlink

Finally it renewed certificate:

2023-10-17 14:38:23,765:DEBUG:certbot._internal.storage:Writing new private key to /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/archive/bjns60jwh9.qa.egnyte-appliance.com-0003/privkey17.pem.
2023-10-17 14:38:23,765:DEBUG:certbot._internal.storage:Writing certificate to /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/archive/bjns60jwh9.qa.egnyte-appliance.com-0003/cert17.pem.
2023-10-17 14:38:23,765:DEBUG:certbot._internal.storage:Writing chain to /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/archive/bjns60jwh9.qa.egnyte-appliance.com-0003/chain17.pem.
2023-10-17 14:38:23,765:DEBUG:certbot._internal.storage:Writing full chain to /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/archive/bjns60jwh9.qa.egnyte-appliance.com-0003/fullchain17.pem.
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var config_dir=/opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com (set by user).
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var work_dir=/opt/certservice/3.3.0.1652419071/LE/work (set by user).
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var logs_dir=/opt/certservice/3.3.0.1652419071/LE/logs (set by user).
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var authenticator=webroot (set by user).
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var webroot_path=/usr/share/nginx/html/letsencrypt (set by user).
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var webroot_path=/usr/share/nginx/html/letsencrypt (set by user).
2023-10-17 14:38:23,777:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2023-10-17 14:38:23,778:DEBUG:certbot._internal.storage:Writing new config /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0003.conf.new.
2023-10-17 14:38:23,780:DEBUG:certbot._internal.display.obj:Notifying user:
Successfully received certificate.
Certificate is saved at: /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0003/fullchain.pem
Key is saved at:         /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0003/privkey.pem
This certificate expires on 2024-01-15.

My web server is (include version): Nginx

The operating system my web server runs on is (include version): CentOS Linux release 7.9.2009 (Core)

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.23.0

3

Hi @April, and welcome to the LE community forum :slight_smile:

-000x extensions are created when a new cert is required; But the default cert name for it conflicts with an already existing cert name.
As an example: That can happen when multiple domains are included into a single cert and then either one name is added or one name is removed.

You can see the list of certs and the names they cover with:

certbot certificates

[please show that output so that we can better explain what happened and what you can best do to correct it (now and in the future)]

3 Likes
4

Thanks @rg305 I came across this post which says that something might be broken. This one as well

I am wondering why it chose 0003 and complained for 1,2 and 4

1 Like
5

Please show us the output of:

certbot certificates

So that we can try to answer that question.

3 Likes
6

I am not sure if this helps

sudo /opt/certservice/3.3.0.1652419071/venv/bin/certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Python 3.6 support will be dropped in the next release of Certbot - please upgrade your Python version.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certificates found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

It seems our cert service has different version upgraded and installed in venv

7

You also need to use all the --config-dir options.

1 Like
9

That's likely part of why you have -000x certificates.
You should not use that unless you know what it does and absolutely have to.

And it seems that something has changed the expected symlinks:

2 Likes
10

@Osiris All the --config-dir options ?

11

Why would it pick up 0003 as the number and complain for 0004 broken ?

12

certbot tries to renew them all.
One number doesn't replace any other number.
If you have:

  • name
  • name-0001
  • name-0002
  • name-0003
  • name-0004

Then you have five certs.
If any of them fail, then those fail [for various possible reasons].
If any can be renewed, then they are renewed.
What shows?:

ls -l /opt/certservice/3.3.0.1645093818/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/
3 Likes
13

--config-dir is the most important, as it is the location where the certificates are stored.

It looks like something or someone has destroyed some of the certificate locations, looking at the symlink errors.

3 Likes
14 
total 4
drwxr-xr-x 2 egnyte egnyte  93 Jan 15  2022 bjns60jwh9.qa.egnyte-appliance.com
drwxr-xr-x 2 egnyte egnyte  93 Feb 15  2022 bjns60jwh9.qa.egnyte-appliance.com-0001
drwxr-xr-x 2 egnyte egnyte  93 Apr 17  2022 bjns60jwh9.qa.egnyte-appliance.com-0002
drwxr-xr-x 2 egnyte egnyte  93 Oct 17 14:38 bjns60jwh9.qa.egnyte-appliance.com-0003
drwxr-xr-x 2 egnyte egnyte  93 Jul 17  2022 bjns60jwh9.qa.egnyte-appliance.com-0004
-rw-r--r-- 1 egnyte egnyte 740 Jan 15  2022 README
15

What the service does is that it TARs everything for a domain and stores the tar as binary blob. When it needs to renew certs, it UNTARs the blob to working dir location and provides that as the --config-dir option to certbot command

16

Since it reported errors for all version other than -0003, should I assume that somehow the links to 0003 were intact and it was able to renew that certificate ?

17

I wouldn't assume anything.

What shows?:

ls -l /opt/certservice/3.3.0.1645093818/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0003
3 Likes
18

Yes, the log is from Oct17 and said new cert was issued. You can see your cert history using a tool like https://crt.sh but know that sometimes there are long lags before certs show up there (>24 hours). See history of this domain (link here)

You could also view the active cert for that domain with something like openssl. This domain is not open to the public internet so I couldn't reach it but presumably you can.

openssl s_client -connect (domain):443

Your Centos 7 if an older openssl may need this to work

openssl s_client -servername (domain) -connect (domain):443  2>/dev/null | openssl x509 -noout -dates .
2 Likes
19

You need to make sure it saves and restores symlinks correctly.

3 Likes
20 
total 4
lrwxrwxrwx 1 egnyte egnyte  64 Oct 17 14:38 cert.pem -> ../../archive/bjns60jwh9.qa.egnyte-appliance.com-0003/cert17.pem
lrwxrwxrwx 1 egnyte egnyte  65 Oct 17 14:38 chain.pem -> ../../archive/bjns60jwh9.qa.egnyte-appliance.com-0003/chain17.pem
lrwxrwxrwx 1 egnyte egnyte  69 Oct 17 14:38 fullchain.pem -> ../../archive/bjns60jwh9.qa.egnyte-appliance.com-0003/fullchain17.pem
lrwxrwxrwx 1 egnyte egnyte  67 Oct 17 14:38 privkey.pem -> ../../archive/bjns60jwh9.qa.egnyte-appliance.com-0003/privkey17.pem
-rw-r--r-- 1 egnyte egnyte 692 May 18  2022 README

And this certificate is actually valid till 15-Jan-24

21

I believe links were infact broken for the certificates it complained for. Though I am not sure why it was able to renew -0003 though. That should have been broken as well.

If all the links are broken, it should have created -0005 ?

sudo /opt/certservice/3.3.0.1652419071/venv/bin/certbot certificates --config-dir=/opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Python 3.6 support will be dropped in the next release of Certbot - please upgrade your Python version.
Renewal configuration file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0001.conf produced an unexpected error: expected /opt/certservice/3.3.0.1642516670/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0001/cert.pem to be a symlink. Skipping.
Renewal configuration file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0002.conf produced an unexpected error: expected /opt/certservice/3.3.0.1645093818/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0002/cert.pem to be a symlink. Skipping.
Renewal configuration file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0004.conf produced an unexpected error: expected /opt/certservice/3.3.0.1649426201/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0004/cert.pem to be a symlink. Skipping.
Renewal configuration file /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com.conf produced an unexpected error: expected /opt/certservice/3.3.0.1641979321/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com/cert.pem to be a symlink. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: bjns60jwh9.qa.egnyte-appliance.com-0003
    Serial Number: 39e971c31114d500564032169685c8065d0
    Key Type: RSA
    Domains: bjns60jwh9.qa.egnyte-appliance.com
    Expiry Date: 2024-01-15 13:38:21+00:00 (VALID: 74 days)
    Certificate Path: /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0003/fullchain.pem
    Private Key Path: /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/live/bjns60jwh9.qa.egnyte-appliance.com-0003/privkey.pem

The following renewal configurations were invalid:
  /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0001.conf
  /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0002.conf
  /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com-0004.conf
  /opt/certservice/3.3.0.1652419071/LE/config/bjns60jwh9.qa.egnyte-appliance.com/renewal/bjns60jwh9.qa.egnyte-appliance.com.conf

You mentioned not to use --force-renewal option unless we are absolutely sure that is the intent. What would it do ?

22

Please read the documentation / user guide if you don't know what certain options actually do. And frankly, don't use options of which you don't know its function to begin with. That's just not very smart.

1 Like