Certbot - Live Symlinks Not Updating As Expected - New Certificate Available But Not Used

I ran this command:
sudo letsencrypt certonly --standalone -w /etc/letsencrypt -d leopold.nessys.nl -d www.leopold.nessys.nl

It produced this output:
IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at
    /etc/letsencrypt/live/leopold.nessys.nl/fullchain.pem. Your cert
    will expire on 2017-05-24. To obtain a new version of the
    certificate in the future, simply run Let’s Encrypt again.

  • If you like Let’s Encrypt, please consider supporting our work by:

    Donating to ISRG / Let’s Encrypt: https://letsencrypt.org/donate
    Donating to EFF: https://eff.org/donate-le

The operating system my web server runs on is (include version):
Ubuntu 16.04

It seems as if letsencrypt gives me exactly the same certificate back which was expired 6 days ago. There were no error messages or anything. Could this be a rights issue or anything? I remember having renewed the certificate already once before without any issues.

Hi @Philiiiiiipp,

Your certificate was successfully renewed:

https://crt.sh/?id=146151173

The trouble is that your server is still using the old certificate rather than the new one. However, the new one exists (almost certainly including on your system), so the question is how to make your web server start using the new one.

If your web server was configured properly (pointing at /etc/letsencrypt/live and not /etc/letsencrypt/archive), it should be possible to reload it (e.g. something like service apache2 reload or service nginx reload) and then pick up the new certificate immediately. If it’s somehow pointing at the files in /etc/letsencrypt/archive, you should change the configuration to point at /etc/letsencrypt/live instead.

But then why does certbot (letsencrypt) itself report that the newly obtained certificate will expire on 2017-05-24? The web server configuration shouldn't influence that output, should it?

I wonder if it's the same issue as this?

If so, @Philiiiiiipp, you may find that the symbolic links in /etc/letsencrypt/live/leopold.nessys.nl/ are pointing to an old version of the certificate, but the newer one is in /etc/letsencrypt/archive/leopold.nessys.nl/ and the symbolic links need to be updated manually.

Hmmm! If that’s so, it would be great to see the logs from /var/log/letsencrypt.

Is it save for me to post the whole logfile here ? I see there are some tokens etc in there.

Here are the last lines:

2017-05-30 12:17:08,282:DEBUG:letsencrypt.storage:Writing new private key to /etc/letsencrypt/archive/leopold.nessys.nl/privkey2.pem.
2017-05-30 12:17:08,283:DEBUG:letsencrypt.storage:Writing certificate to /etc/letsencrypt/archive/leopold.nessys.nl/cert2.pem.
2017-05-30 12:17:08,283:DEBUG:letsencrypt.storage:Writing chain to /etc/letsencrypt/archive/leopold.nessys.nl/chain2.pem.
2017-05-30 12:17:08,283:DEBUG:letsencrypt.storage:Writing full chain to /etc/letsencrypt/archive/leopold.nessys.nl/fullchain2.pem.
2017-05-30 12:17:08,284:DEBUG:letsencrypt.storage:Writing new config /etc/letsencrypt/renewal/leopold.nessys.nl.conf.new.
2017-05-30 12:17:08,287:INFO:letsencrypt.reporter:Reporting to user: Congratulations! Your certificate and chain have been saved at /etc/letsencrypt/live/leopold.nessys.nl/fullchain.pem. Your cert will expire on 2017-05-24. To obtain a new version of the certificate in the future, simply run Let's Encrypt again.
2017-05-30 12:17:08,287:INFO:letsencrypt.reporter:Reporting to user: If you like Let's Encrypt, please consider supporting our work by:

Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
Donating to EFF:                    https://eff.org/donate-le

what is interesting though is that as @jmorahan suggested, the link in the live directory points to an old file

root@leopold:/etc/letsencrypt/live/leopold.nessys.nl# ls -al
lrwxrwxrwx 1 root          root     46 May 30 14:17 cert.pem -> ../../archive/leopold.nessys.nl-0001/cert1.pem
lrwxrwxrwx 1 root          root     47 May 30 14:17 chain.pem -> ../../archive/leopold.nessys.nl-0001/chain1.pem
lrwxrwxrwx 1 root          root     51 May 30 14:17 fullchain.pem -> ../../archive/leopold.nessys.nl-0001/fullchain1.pem
lrwxrwxrwx 1 root          root     49 May 30 14:17 privkey.pem -> ../../archive/leopold.nessys.nl-0001/privkey1.pem 

And the archive folder contains 2 folders with ceritficates, -0001 containing the old one and leopold.nessys.nl containing the new one + some older ones.

root@leopold:/etc/letsencrypt/archive# ls -al
leopold.nessys.nl
leopold.nessys.nl-0001

The archive folder contains:

root@leopold:/etc/letsencrypt/archive/leopold.nessys.nl# ls -alr
total 72
-rw-r--r-- 1 root          root   1704 Dec  2 15:11 privkey4.pem
-rwxrwxrwx 1 root docker 1704 Nov  7  2016 privkey3.pem
-rwxrwxrwx 1 root docker 1704 May 30 14:17 privkey2.pem
-rwxrwxrwx 1 root docker 1704 Oct 23  2016 privkey1.pem
-rw-r--r-- 1 root          root   3517 Dec  2 15:11 fullchain4.pem
-rwxrwxrwx 1 root docker 3489 Nov  7  2016 fullchain3.pem
-rwxrwxrwx 1 root docker 3481 May 30 14:17 fullchain2.pem
-rwxrwxrwx 1 root docker 3448 Oct 23  2016 fullchain1.pem
-rw-r--r-- 1 root          root   1647 Dec  2 15:11 chain4.pem
-rwxrwxrwx 1 root docker 1647 Nov  7  2016 chain3.pem
-rwxrwxrwx 1 root docker 1647 May 30 14:17 chain2.pem
-rwxrwxrwx 1 root docker 1647 Oct 23  2016 chain1.pem
-rw-r--r-- 1 root          root   1870 Dec  2 15:11 cert4.pem
-rwxrwxrwx 1 root docker 1842 Nov  7  2016 cert3.pem
-rwxrwxrwx 1 root docker 1834 May 30 14:17 cert2.pem
-rwxrwxrwx 1 root docker 1801 Oct 23  2016 cert1.pem

Linking the files in the live folder to the newly created files gives me the latest certificate now. It seems to me as if some folders got created/mixed up in the creation of the new certificate.

One alarming thing here is that cert2.pem (etc.) look newer than cert3.pem and cert4.pem. That should not happen (in addition to the fact that the live symlinks should always point to the newest version).

I’m also wondering about the fact that version of each file is owned by root:root, while the older ones are owned by docker:docker. Is it possible that something changed about how you ran Certbot with the result that it had different permissions to write to these directories at different times?

I changed the group of the leopold.nessys.nl folder and the underlying files a couple of month back because I thought there was a rights issue with the certificates and the docker container (turns out this was not the case) the owner is root though.

I always used the same command, the only change being that I have added the second www.leopold.nessys.nl a couple of month back. I am quite sure about that because I am always running this from a shell script.

sudo letsencrypt certonly --standalone -w /etc/letsencrypt -d leopold.nessys.nl -d www.leopold.nessys.nl

live/leopold.nessys.nl should have symlinks to archive/leopold.nessys.nl, not archive/leopold.nessys.nl-0001

Could it be that Certbot simply uses (old symlink target + 1) without verifying that the symlinks pointed to the correct archive directory?

The alternatives would, i suppose, be checking the symlinks and raising an exception if they’re wrong, or listing the (correct) archive directory to find the highest numbered files (or perhaps the lowest unused number).

How much effort you should take to handle corrupt data is a question, but this situation is not unheard of. Usually because someone renamed the live directory to remove “-0001” but didn’t know they had to fix the symlinks (or renewal file).

:confused:

@mnordhoff, thanks for catching that!

@Philiiiiiipp, did you previously update the symlinks manually for some other reason?

Yes, the symlink target is regarded as defining what is the correct archive directory.

Hmm, no I don’t think so. So far it always worked without a problem. This is also not the only server I am using letsencrypt on, but the only one which has this problem.

I will check the logs tomorrow to see if I see anything out of the ordinary.

Following up here from the related issue Certbot - Certificate Issued from Let’s Encrypt but Live Certificates Pointing At Previous Versions, I’m unable to reproduce this (without manually modifying /etc/letsencrypt).

@Philiiiiiipp, if you didn’t find anything particularly interesting in the logs yourself, would you be willing to share them with me so I can take a look? Certbot currently doesn’t do much to try and help you if you incorrectly modify the files in /etc/letsencrypt, but if Certbot configured the files this way themselves, it’s a serious bug that I’d like to resolve.

Ideally what I’d like is the full contents of /var/log/letsencrypt. The only potentially sensitive values in there are domains, e-mail, and IP addresses. Feel free to redact those as you feel is appropriate. Additionally, you can e-mail them to me directly if you prefer at bmw@eff.org.

Hey,

ok I send it to you in your email.

Thanks for the logs.

Looking through them, I’m confident /etc/letsencrypt was modified by something other than Certbot in the last few months. It looks like what happened is you removed a domain that you were previously obtaining a certificate for (for your reference, this happened in letsencrypt.log.3). This is what caused leopold.nessys.nl-0001 to be created.

At the end of that log, the certificate lineage for leopold.nessys.nl-0001 is created, files are properly written to the archive as /etc/letsencrypt/archive/leopold.nessys.nl-0001/<filename>1.pem, and at least /etc/letsencrypt/live/leopold.nessys.nl-0001/fullchain.pem is a symlink to those files. Unfortunately, the status of any of the other symlinks is not included in the logs.

In the next log (letsencrypt.log.2), the files for leopold.nessys.nl-0001 are corrupted. Specifically, /etc/letsencrypt/live/leopold.nessys.nl-0001/cert.pem isn’t a symlink (or doesn’t exist). Additionally, all symlinks in /etc/letsencrypt/live/leopold.nessys.nl point to /etc/letsencrypt/archive/leopold.nessys.nl-0001. I’m not sure how this happened, but I’m confident in saying this wasn’t caused by letsencrypt. It seems very unlikely that only /etc/letsencrypt/live/leopold.nessys.nl-0001/cert.pem was corrupted and the last time the files in /etc/letsencrypt/live/leopold.nessys.nl were modified (letsencrypt.log.5), /etc/letsencrypt/live/leopold.nessys.nl/fullchain.pem pointed to /etc/letsencrypt/archive/leopold.nessys.nl/fullchain4.pem.

When the files in leopold.nessys.nl-0001 were created a little over 3 months ago, did you modify /etc/letsencrypt so you could continue to use the symlinks in /etc/letsencrypt/live/leopold.nessys.nl/ rather than modifying your scripts and/or web server’s configuration to use /etc/letsencrypt/live/leopold.nessys.nl-0001? This is my best explanation about what happened.

The easiest way to fix the problem is to delete /etc/letsencrypt (or move it somewhere else) and run your script again. You will obtain a new certificate despite you already having a valid one, but this should give you a clean state with a valid configuration in the /etc/letsencrypt directory.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.