certbot.errors.CertStorageError: expected /etc/letsencrypt/live/vriezon-tel.werkonderweg.nl/fullchain.pem to be a symlink

Help
1

I ran this command:
certbot renew
It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/customerdomain.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 68, in _reconstitute
    renewal_candidate = storage.RenewableCert(full_path, config)
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 463, in __init__
    self._check_symlinks()
  File "/usr/lib/python3/dist-packages/certbot/storage.py", line 522, in _check_symlinks
    "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/customerdomain/fullchain.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/customerdomain.conf is broken. Skipping.

My web server is (include version):
nginx/1.14.2

The operating system my web server runs on is (include version):
Debian 10

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.31.0

Hello,

Somehow the symlink from the fullchain.pem file gets removed and i'm not sure why. This is the result ls -l on the live folder:

total 8
-rw-r--r-- 1 root root  692 Feb 17  2021 README
lrwxrwxrwx 1 root root   51 Nov 15 10:19 cert.pem -> ../../archive/customerdomain/cert4.pem
lrwxrwxrwx 1 root root   52 Nov 15 10:19 chain.pem -> ../../archive/customerdomain/chain4.pem
-rw-r--r-- 1 root root 3706 Oct 14 03:44 fullchain.pem
lrwxrwxrwx 1 root root   54 Nov 15 10:19 privkey.pem -> ../../archive/customerdomain/privkey4.pem

I restored the symlink using
rm fullchain.pem
ln -s ../../archive/customerdomain/fullchain4.pem fullchain.pem

This seemed to fix the issue and i could run certbot renew again. But this issue has happened at multiple customer servers right now, so I need to figure out why this is happening. The servers are used for 3CX VoIP systems. The 3CX NGINX service is using the certificates from the /etc/letsencrypt/live folder.
Could this be an rights issue on the live and archive folders? What should those rights be?

Thank you

2

I guess not. Your nginx also uses the privkey.pem and it was not affected.

Did you use a "hook" when first creating the cert? Can you post the (redacted) contents of the conf file in /etc/letsencrypt/renewal for this domain?

Also, does your nginx server config reference the files in the .../live/customerdomain/ folder or from somewhere else?

1 Like
3 
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/customerdomain.nl
cert = /etc/letsencrypt/live/customerdomain.nl/cert.pem
privkey = /etc/letsencrypt/live/customerdomain.nl/privkey.pem
chain = /etc/letsencrypt/live/customerdomain.nl/chain.pem
fullchain = /etc/letsencrypt/live/customerdomain.nl/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
account = *redacted*
renew_hook = systemctl reload nginx

My nginx config file references to certificatefiles inside the 3CX folder itself. The 3CX install folder has certificate files inside which are created using a symlink to the letsencrypt/live folder.

5

Your certbot renewal conf looks clean. You should review whatever sets up that symlink in 3CX folder. Was it a one-time thing? Or, is it done on a schedule or restart? Did it not like making a symlink to another symlink and replaced the /live/ symlink with a file?

I do not have any other ideas than to check that. Your certbot looks very typical yet something unusual happens. So, probably not in certbot. nginx can follow multiple symlinks and does for your privkey so no reason to do that.

2 Likes
6

The symlinks in the 3CX folder are created by an bash script, this is a 1 time only command when setting up certbot. It has been working fine for already 4 renewals, but since a short time it's throwing above error on a few of our servers, not all. However, all the servers are identical regarding certbot, nginx and 3cx configuration.
In the bash script I did notice 2 rules which edit the rights on the /etc/letsencrypt/{live,archive} folders and i've got a feeling that those steps are not necessary since i'm using symlinks.

chmod 0750 /etc/letsencrypt/{live,archive}
chgrp phonesystem /etc/letsencrypt/{live,archive}

Creating the symlinks:

ln -s /etc/letsencrypt/live/$domain_name/fullchain.pem /var/lib/3cxpbx/Bin/nginx/conf/Instance1/$domain_name-crt.pem
ln -s /etc/letsencrypt/live/$domain_name/privkey.pem /var/lib/3cxpbx/Bin/nginx/conf/Instance1/$domain_name-key.pem
echo "Creating symlinks..."
sleep 5
7

Maybe not. Those only affect those folders not each folder/files within. It depends what authority nginx runs as. I'll leave that for you to sort - or someone else.

But, there must be something else happening as only fullchain is affected and not privkey. Certbot acts on all /live files as a set and what you show is also consistent between these two.

Your certbot renewal conf is a little unusual in that it uses standalone authenticator but also reloads nginx after renewing certs. So, I assume your Voip uses ports other than 80 to allow this to work. We don't often see standalone with a viable nginx but I don't see a problem in specialized cases. I do not see how this would affect the certs though - certbot still acts on them all as a set.

I do not know what more to advise other than to point this out. Sorry.

1 Like
9

You need to delete that file and reissue the cert.
[which should remake the required symbolic link]

And then review your entire process - which may at some point have the src dst for creating a symlink reverse OR a bad cp src dst.

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.