Certbot renew - symlink fail error

Please advise , my ssl is going to expire by tommorrow.

I tried to renew my ssl cert, using certbot renew command , it returns error ""expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/sprigsys.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/sprigsys.com.conf is broken. Skipping.

Error message

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 68, in _reconstitute
renewal_candidate = storage.RenewableCert(full_path, config)
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 463, in init
self._check_symlinks()
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 522, in _check_symlinks
"expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/sprigsys.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/sprigsys.com.conf is broken. Skipping.

My domain is: sprigsys.com

I ran this command: certbot renew also tried certbot update_symlinks

It produced this output: error "expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/sprigsys.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/sprigsys.com.conf is broken. Skipping.

My web server is (include version): NGNIX (docker container)

The operating system my web server runs on is (include version):linux

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0

Tree output:
tree /etc/letsencrypt
/etc/letsencrypt
├── accounts
│ └── acme-v02.api.letsencrypt.org
│ └── directory
│ └── f9bc1df3fa5ee6536ee78c1ef68babe4
│ ├── meta.json
│ ├── private_key.json
│ └── regr.json
├── archive
│ └── sprigsys.com
│ ├── cert1.pem
│ ├── chain1.pem
│ ├── fullchain1.pem
│ └── privkey1.pem
├── cli.ini
├── csr
│ ├── 0000_csr-certbot.pem
│ ├── 0001_csr-certbot.pem
│ ├── 0002_csr-certbot.pem
│ ├── 0003_csr-certbot.pem
│ ├── 0004_csr-certbot.pem
│ ├── 0005_csr-certbot.pem
│ ├── 0006_csr-certbot.pem
│ ├── 0007_csr-certbot.pem
│ ├── 0008_csr-certbot.pem
│ ├── 0009_csr-certbot.pem
│ ├── 0010_csr-certbot.pem
│ ├── 0011_csr-certbot.pem
│ ├── 0012_csr-certbot.pem
│ ├── 0013_csr-certbot.pem
│ ├── 0014_csr-certbot.pem
│ ├── 0015_csr-certbot.pem
│ ├── 0016_csr-certbot.pem
│ └── 0017_csr-certbot.pem
├── keys
│ ├── 0000_key-certbot.pem
│ ├── 0001_key-certbot.pem
│ ├── 0002_key-certbot.pem
│ ├── 0003_key-certbot.pem
│ ├── 0004_key-certbot.pem
│ ├── 0005_key-certbot.pem
│ ├── 0006_key-certbot.pem
│ ├── 0007_key-certbot.pem
│ ├── 0008_key-certbot.pem
│ ├── 0009_key-certbot.pem
│ ├── 0010_key-certbot.pem
│ ├── 0011_key-certbot.pem
│ ├── 0012_key-certbot.pem
│ ├── 0013_key-certbot.pem
│ ├── 0014_key-certbot.pem
│ ├── 0015_key-certbot.pem
│ ├── 0016_key-certbot.pem
│ └── 0017_key-certbot.pem
├── live
│ └── sprigsys.com
│ ├── README
│ ├── cert.pem
│ ├── chain.pem
│ ├── fullchain.pem
│ └── privkey.pem
├── renewal
│ └── sprigsys.com.conf
└── renewal-hooks
├── deploy
├── post
└── pre

FULL ERROR Message

$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/sprigsys.com.conf


Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 68, in _reconstitute
renewal_candidate = storage.RenewableCert(full_path, config)
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 463, in init
self._check_symlinks()
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 522, in _check_symlinks
"expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/sprigsys.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/sprigsys.com.conf is broken. Skipping.


No renewals were attempted.

Additionally, the following renewal configurations were invalid:
/etc/letsencrypt/renewal/sprigsys.com.conf (parsefail)


0 renew failure(s), 1 parse failure(s)
$

1 Like

Welcome to the Let's Encrypt Community, Amit :slightly_smiling_face:

Please run:

sudo certbot update_symlinks

then try to renew again.

1 Like

Hi @Amit

your question has your complete answer. You have changed these files manual (that's wrong), so you have created that error. Undo that.

If you don't know how: Your system has a documentation - use it.

Tried it , no luck...
$ sudo certbot update_symlinks
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Expected /etc/letsencrypt/live/sprigsys.com/privkey.pem to be a symlink

1 Like

@JuergenAuer -

Please note i am too new to this, any help from your end will save my life...
May be later we can look at the cause of it....

1 Like

@JuergenAuer @griffin

Have made some progress...
Here is the latest error, little guidance from your end will save my life...
$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/sprigsys.com.conf


Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')
Attempting to renew cert (sprigsys.com) from /etc/letsencrypt/renewal/sprigsys.com.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.'). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/sprigsys.com/fullchain.pem (failure)


All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/sprigsys.com/fullchain.pem (failure)


1 renew failure(s), 0 parse failure(s)

1 Like

The files in /live/sprigsys.com/ should be symbolic links to the most recent and corresponding files in ../../archive/sprigsys.com/.

What's the output of ls -l /etc/letsencrypt/live/sprigsys.com/ ?

1 Like

$ cd /etc/letsencrypt/live/sprigsys.com/
$ ls -l
total 4
-rw-r--r-- 1 root root 740 Jan 2 15:56 README
lrwxrwxrwx 1 root root 36 Mar 30 19:11 cert.pem -> ../../archive/sprigsys.com/cert1.pem
lrwxrwxrwx 1 naveen naveen 37 Mar 30 19:20 chain.pem -> ../../archive/sprigsys.com/chain1.pem
lrwxrwxrwx 1 naveen naveen 41 Mar 30 19:21 fullchain.pem -> ../../archive/sprigsys.com/fullchain1.pem
lrwxrwxrwx 1 naveen naveen 39 Mar 30 19:22 privkey.pem -> ../../archive/sprigsys.com/privkey1.pem

1 Like

On my system everything is owned by root. Could this be the issue? Although I would think root can access the file(s), even if it's owned by a different user.. As it's root we're talking about. What's the output of ls -l /etc/letsencrypt/archive/sprigsys.com/ ? To see the owner. Note that the contents of privkey.pem should be kept safe, but ls -l can't hurt :slight_smile:

1 Like

Here is the output:
$ cd /etc/letsencrypt/archive/sprigsys.com/
$ ls -l
total 16
-rw-r--r-- 1 root root 1858 Mar 30 18:25 cert1.pem
-rw-r--r-- 1 root root 1586 Mar 30 18:25 chain1.pem
-rw-r--r-- 1 root root 3444 Mar 30 18:25 fullchain1.pem
-rwxrwxrwx 1 root root 1704 Mar 30 18:25 privkey1.pem

1 Like

Then you should never change such files manual. And you should always create a backup before changing things.

@JuergenAuer -
Noted, by the way it was done by a ex employee and I am unlucky fellow who is assigned to clean it up.

1 Like

Although the permissions of privkey1.pem are a little bit weird, I don't see anything why certbot would need to complain about it?

1 Like

@JuergenAuer

Any suggestions from your much appreciated.

1 Like

@Osiris

What do you suggest in this case?

1 Like

You could try to re-do the symbolic links manually:

cd /etc/letsencrypt/live/sprigsys.com/
rm *.pem
ln -s ../../archive/sprigsys.com/cert1.pem cert.pem
ln -s ../../archive/sprigsys.com/chain1.pem chain.pem
ln -s ../../archive/sprigsys.com/fullchain1.pem fullchain.pem
ln -s ../../archive/sprigsys.com/privkey1.pem privkey.pem
2 Likes

@Osiris

  1. I have already tried the manual linking step already.
  2. Do you think it's a certbot BUG??
  3. Here is the conf file, hope this helps you to provide the expert opinion.
    image
1 Like

Probably not, but version 0.31.0 is quite old, so to exclude a bug in an older version you might want to update to a newer version (which probably requires installing snap).

Nothing strange there unfortunately.

If I had this issue, I'd probably dive into the certbot source code and debug it from the code directly. However, that's quite difficult without Python and/or Linux experience I'm afraid.

1 Like

@Osiris

What's your suggestion?

1 Like

As a last resort, you could always back up the /live/, /archive/ and /renewal/ directory (just in case) and remove those directories from /etc/letsencrypt/ and start over.

1 Like