Unable to renew certificates with certbot renew inside docker container

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: ivcappservers.com

I ran this command: certbot renew

It produced this output: Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/ivcappservers.com.conf

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/_internal/renewal.py", line 71, in _reconstitute
renewal_candidate = storage.RenewableCert(full_path, config)
File "/usr/lib/python3/dist-packages/certbot/_internal/storage.py", line 471, in init
self._check_symlinks()
File "/usr/lib/python3/dist-packages/certbot/_internal/storage.py", line 537, in _check_symlinks
raise errors.CertStorageError(
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/ivcappservers.com/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/ivcappservers.com.conf is broken. Skipping.

No renewals were attempted.

Additionally, the following renewal configurations were invalid:
/etc/letsencrypt/renewal/ivcappservers.com.conf (parsefail)

0 renew failure(s), 1 parse failure(s)

Additional Info: I manually got the certs by installing certbot inside the nginx container, and having volumes to make sure they persisted beyond the container. For whatever reason the symbolic links are not there.

Output of certbot update_symlinks: Saving debug log to /var/log/letsencrypt/letsencrypt.log
Expected /etc/letsencrypt/live/ivcappservers.com/cert.pem to be a symlink

My web server is (include version): Nginx(nginx:latest)

The operating system my web server runs on is (include version): Ubuntu 20.04, using Docker

My hosting provider, if applicable, is: Linode

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.12.0

Two questions:

  1. Is there a way to accomplish this without the symbolic links?
  2. If not, is there a way to do this using just the certs, or do I have to just request certs all over again?

Color me lost and confused :slightly_smiling_face:

2

That is a very strange failure.
Did you somehow copy files to that path?
Let's have a look at:
ls -l /etc/letsencrypt/live/

1 Like
3

On the host I have 2 volumes created in the docker-compose:
volumes:
- ./letsencrypt:/etc/letsencrypt
- ./nginx-html:/usr/share/nginx/html
So that the information is persisted beyond the containers lifetime. My suspicion is that the symbolic link doesn't survive the volume mapping, but I'm not sure.

The output on the host and inside the docker container is the same
ls -l /etc/letsencrypt/live/
total 8
-rw-r--r-- 1 1000 1000 740 Jan 6 15:59 README
drwxr-xr-x 2 1000 1000 4096 Jan 6 15:59 ivcappservers.com

root@ab334e658fa4:/# ls -l /etc/letsencrypt/live/ivcappservers.com/
total 24
-rw-r--r-- 1 1000 1000 692 Jan 6 15:59 README
-rw-r--r-- 1 1000 1000 1850 Jan 6 15:59 cert.pem
-rw-r--r-- 1 1000 1000 3749 Jan 6 15:59 chain.pem
-rw-r--r-- 1 1000 1000 5599 Jan 6 15:59 fullchain.pem
-rw------- 1 1000 1000 1704 Jan 6 15:59 privkey.pem

4

Those should all be symlinks (to the latest files within the archive folder).

1 Like
5

In this case, they can't be because they're mapped to the host, and no link would exist because it's inside a container which can be destroyed and recreated.

  1. Can they manually be restored as symbolic links to appease certbot, or
  2. I have no idea why this has to be so convoluted, there must be a simple way to get certs when they're only trying to prove domain ownership. A method that isn't so fragile, IMO.

Either way, you accurately described the issue as to why it won't renew, I just need a solution.

6

I don't understand.. The certificates and would-be symlinks are on persistent storage, right? And symbolic links are just elements in the data structure of a file system capable of refering relatively to other files.. So I don't really see the problem?

2 Likes
7

I'd start by deleting the files [move them if they don't actually exist in the /archive/ folder].
Then recreate the symlinks.

1 Like
8

The directories are mapped, but the paths will be different, so a symbolic link would point to nowhere on the host.

However, the fact that they're NOT symbolic links and nothing was different or changed in the process of getting the certificate by definition presents a problem.

9

I'll try that later today and let you know the result, it sounds like it's the best course of action to try.

10

One thing I took from your comment, and which I'll try as soon as I can, is that they symbolic links should be consistent from host to container.

BUT ONLY if the paths are the same, so I'm going to make the paths mirror each other as well and see if that helps.

11

I did exactly what you said, and it worked a treat!!!

certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/ivcappservers.com.conf

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Account registered.
Simulating renewal of an existing certificate for ivcappservers.com
Performing the following challenges:
http-01 challenge for ivcappservers.com
Using default address 80 for authentication.
Waiting for verification...
Cleaning up challenges

new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/ivcappservers.com/fullchain.pem

Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/ivcappservers.com/fullchain.pem (success)

Thanks for everything, it's very much appreciated!

1 Like
12

I'm really glad you replied, it made me think!

Now that I've mirrored the paths exactly, the symbolic links survive destruction and creation of the containers!

I was able to simulate renewal even after i destroyed the docker-compose.

2 Likes
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.