CertStorageError: expected /etc/letsencrypt/live/example.com/cert.pem to be a symlink

I’m using nginx. The certs work fine. Extracted error report from log created by update:

CertStorageError: expected /etc/letsencrypt/live/example.com/cert.pem to be a symlink

Sure, I can understand that, but I have no idea where to move the certs in the /live/ directory to. I prefer to keep them in the /etc/letsencrypt/ directory structure. Is there any proper way to do this or do I just pick any directory name and then create the symlinks from /live/ to /my_directory/ ?

FWIW I followed the following tutorial and I’m running Debian 8 Jessie with nginx not Apache. My server blocks all work correctly except I think the most recent update request (set to weekly in a cron job) seems to have killed one of my domains.

tutorial: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-debian-8

Why do you get the error message in the first place? Did you manually change things in the /etc/letsencrypt directory tree? Because the tutorial you’re linking to doesn’t ask you to do that…

Certbot effectively assigns a special meaning to the filenames and structure within /etc/letsencrypt and there isn't currently a reliable way to reorganized it without risking confusing Certbot about the status of your certificates.

I accepted the default installation. I changed nothing in the /etc/letsencrypt/ directory. I followed the tutorial but it had no instructions other than to let the installation proceed automatically. It had no instructions to move any certificates or change locations. None were moved nor any locations changed.

The error arises when my cron job requests renewals of the certificates. This is the cron command:

/usr/bin/certbot renew --noninteractive --renew-hook “/bin/systemctl reload nginx” >> /var/log/le-renew.log

I am merely trying to appease certbot by removing the reason for its error messages that indicate /live/ should contain symlinks, not the certificates themselves. I can quote the log for one certificate if anybody wishes.

Thanks for your help. :slight_smile:

Might as well save time by quoting the log after removing the redundant entries for all but one site. (Edited to change URL to example.com.)

2017-11-20 12:26:45,261:DEBUG:certbot.main:Root logging level set at 30
2017-11-20 12:26:45,262:INFO:certbot.main:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-11-20 12:26:45,262:DEBUG:certbot.main:certbot version: 0.10.2
2017-11-20 12:26:45,262:DEBUG:certbot.main:Arguments: [’-q’]
2017-11-20 12:26:45,263:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2017-11-20 12:26:45,264:WARNING:certbot.renewal:expected /etc/letsencrypt/live/example.com/cert.pem to be a symlink
2017-11-20 12:26:45,264:WARNING:certbot.renewal:Renewal configuration file /etc/letsencrypt/renewal/example.com.conf is broken. Skipping.
2017-11-20 12:26:45,265:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File “/usr/lib/python2.7/dist-packages/certbot/renewal.py”, line 59, in _reconstitute
renewal_candidate = storage.RenewableCert(full_path, config)
File “/usr/lib/python2.7/dist-packages/certbot/storage.py”, line 392, in init
self._check_symlinks()
File “/usr/lib/python2.7/dist-packages/certbot/storage.py”, line 431, in _check_symlinks
"expected {0} to be a symlink".format(link))
CertStorageError: expected /etc/letsencrypt/live/example.com/cert.pem to be a symlink

2017-11-20 12:26:45,281:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.10.2’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 849, in main
return config.func(config, plugins)
File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 655, in renew
renewal.handle_renewal_request(config)
File “/usr/lib/python2.7/dist-packages/certbot/renewal.py”, line 430, in handle_renewal_request
len(renew_failures), len(parse_failures)))
Error: 0 renew failure(s), 7 parse failure(s)

It has occurred to me that the file in /etc/letsencrypt/renewal may be relevant. Here it is minus redacting for reasons of privacy:

version = 0.10.2
archive_dir = /etc/letsencrypt/archive/example.com
cert = /etc/letsencrypt/live/example.com/cert.pem
privkey = /etc/letsencrypt/live/example.com/privkey.pem
chain = /etc/letsencrypt/live/example.com/chain.pem
fullchain = /etc/letsencrypt/live/example.com/fullchain.pem

[renewalparams]
authenticator = webroot
installer = None
account = 99f3904cac84951a2636e98904be5dc7
webroot_path = /home/example.com,
[[webroot_map]]
example.com = /home/example.com
www.example.com = /home/example.com

I just noticed another anomaly. In the /letsencrypt/renewal/ directory the -0001.conf file contains similar to the following:

version = 0.10.2
archive_dir = /etc/letsencrypt/archive/example.net-0001
cert = /etc/letsencrypt/live/example.net-0001/cert.pem
privkey = /etc/letsencrypt/live/example.net-0001/privkey.pem
chain = /etc/letsencrypt/live/example.net-0001/chain.pem
fullchain = /etc/letsencrypt/live/example.net-0001/fullchain.pem

[renewalparams]
authenticator = webroot
installer = None
account = 99f3904cac84951a2636e98904be5dc7
webroot_path = /home/example.net,
[[webroot_map]]
example.net = /home/example.net

There is no example.net-0001 directory in /lets/encrypt/live/ … the paths are void.

That is definitely weird. It sounds like someone or some other software changed the contents of items in /etc/letsencrypt, or else perhaps Certbot crashed and left that directory in an inconsistent state (which is not a usual occurrence but is theoretically possible).

Can you show us what

ls -l /etc/letsencrypt/{archive,live}/*

looks like now?

1 Like

ls -l /etc/letsencrypt/live/
total 36K
drwxr-xr-x 9 root root 4.0K Nov 16 21:48 .
drwxr-xr-x 8 root root 4.0K Nov 15 23:42 ..
drwxr-xr-x 2 root root 4.0K Nov 15 23:42 example.com
drwxr-xr-x 2 root root 4.0K Nov 15 23:42 example.net
drwxr-xr-x 2 root root 4.0K Nov 16 21:48 another-site.com
drwxr-xr-x 2 root root 4.0K Nov 15 23:42 yet-another.com

ls -l /etc/letsencrypt/archive/
total 36K
drwxr-xr-x 9 root root 4.0K Nov 16 21:48 .
drwxr-xr-x 8 root root 4.0K Nov 15 23:42 ..
drwxr-xr-x 2 root root 4.0K Nov 15 23:41 example.com
drwxr-xr-x 2 root root 4.0K Nov 15 23:41 example.net
drwxr-xr-x 2 root root 4.0K Nov 15 23:41 another-site.com
drwxr-xr-x 2 root root 4.0K Nov 16 21:48 yet-another.com

It seems to me that /live/ should have symlinks, perhaps to /archive/ ... maybe I should just delete the files and put in symlinks.

I wish I had an example or explanation of the correct organization of /etc/letsencrypt/.

Sorry, I meant with the literal * at the end so that we also see the contents of the subdirectories!

Thanks for the help schoen. I guess I missed the /*.

However I have rushed in where wise men fear to tread, and attempted an intuitive repair. After 40+ years of software experience sometimes I just intuit solutions. I analyzed what was missing and gave certbot what it appeared to be asking me for.

For each directory in /live/ I replaced the files (e.g. cert.pem, chain.pem…) with symlinks to e.g. cert1.pem, chain1.pem… in /archive/{same name directory}. I ran the certbot command and it complained about the -0001 structure missing. So I copied the directory same name without the -0001 in archive to make an -0001 directory, and then in /live/ I made the -0001 directory and created symlinks as described above to the -0001 directory in /archive/.

I ran the certbot command and … no errors! :slight_smile:

None of my certs are 30 days old yet so I can’t tell if all is in order, but at least the cron job is trowing no errors into the renewal log.

I know it sounds crazy but I think I intuited my way into fixing the problem. I’ll know only for sure in about 3 weeks when the 30 day initial issuance is capable of being renewed and see what the renewal log says.

I’ll appreciate your opinion and/or comments, but it appears I have solved my initial complaint that my renewal log showed errors. As such it appears the issue may have been resolved since running the certbot command no longer throws errors! :slight_smile:

1 Like

That should work! Good intuition.

Thank you very much for your helpful questions which prodded me along the lines of productive conjecture. Many people do not realize that a good question can be equally as helpful as a good answer! It is also helpful knowing there is somebody there to help me, encouraging me to continue working on the problem after I had practically given up.

It appears that things are now in order (throwing no warnings) and that there is nothing further I can do until the certs are old enough to be eligible for renewal. If there is any problem with that I will start a new topic.

Before I finish I have a few final comments:

1.) I’m setting up a new server (just a couple weeks from a fresh, non-configured image) and have followed not only my own notes on how to set up Debian 8 Jessie etc. and nginx, but have had a few problems setting up other services that I have put aside for the time. What I mean is that I’ve been doing massive setup and any of these steps may have caused the problem I’ve experienced. — As a result I wish to state that I do not blame the “Install Let’s Encrypt” tutorial for this problem, and as far as I know it is a good tutorial. I do not hold the author responsible for creating this problem. In all likelihood one of the many failed package installs I’ve been trying may have caused my problem.

2.) One thing I’m critical regarding the article is that the author suggests setting up a cron job to automate renewal and gave the example to run the update script once a day. I am keenly aware that Let’s Encrypt is running on a small budget and in my opinion asking for renewal daily creates a needlessly wasteful use of the Let’s Encrypt certificate renewal system and server.

Instead I set my cron job to run every Sunday.

30 2 * * 0 /usr/bin/certbot renew --noninteractive --renew-hook “/bin/systemctl reload nginx” >> /var/log/le-renew.log

The interpretation of “30 2 * * 0” means run this job every Sunday at 0230.

Digital Ocean is a fairly popular website and if 10,000 readers set their cron to run daily that’s 300,000 queries per month. If they set their cron as I did then that would be 43,000 queries per month, saving Let’s Encrypt 257,000 renewal queries per month!

Let’s Encrypt is presenting an amazing service, and towards a worthy goal, “Let’s encrypt the whole Internet!” I suggest requesting cert renewals no more frequently than weekly would be helpful to the Let’s Encrypt project by using services lightly!

Thank you again for the great help! If I have no more problems perhaps I can participate by helping those with less experience than I.

With that note I consider that this topic is solved and closed. Thank you!

That's a great idea to help take a load off of Let's Encrypt, but the "official" recommendation is actually to run it twice a day, at off-standard times (e.g. 03:12 and 17:38 instead of 03:00 and 15:00). Certbot, by default, will only attempt to renew a certificate if it's within 30 days of expiration. Otherwise, certbot will determine there are no certificates in need of renewal and close without communicating with Let's Encrypt. The twice-a-day standard is to give the best chance at evading and sporadic or regularly-time system failures that can break the renewal process. Don't worry, you're not hammering Let's Encrypt with requests, just be sure you don't also use the --force-renewal or --renew-by-default flags!

Thank you for the feedback. To be honest I have no idea how much resources I use by requesting renewals. Perhaps I’ll return to the article’s suggestion and request daily.

But still, your cert is good for 3 months and I don’t see the point in asking for it so frequently. No matter how much bandwidth you use.

I’ll consider setting it for daily.

It doesn’t use any bandwidth at all, there’s not a single network connection made. It just uses a minuscule amount of processing power on your machine - enough for certbot to read the certificate files it knows about and determine that they’re not ready for renewal. The idea is to give plenty of opportunities to succeed, so that it would take a large number of intermittent failures before your cert expires unrenewed. (Twice-daily fault-tolerance is about sixty failures instead of four.) It all feeds back into “make it as automated as possible.”

1 Like

Thank you for the explanation jared. I’m VERY new to Let’s Encrypt, got my server about a month ago, not my first server but my first server to use SSL, so I never bothered learning about certs (except what they mean in my browser) and only became really interested when I dropped my shared hosting account and had to install postfix-dovecot to support my email. You simply cannot send an email to a Gmail account without a SSL cert, and after studying options I was amazed that Let’s Encrypt offers them for free! That’s particularly considering other organizations that sell you the exact same level of certificate. (One seller told me, yeah but they are good for only 3 months. LOL, guess he never heard of cron jobs!) Anyway I appreciate this project so much that I want to use it well. I had mistakenly thought the update process occupied the project’s bandwidth and I figured at least I could be economical in my use of resources.

I understand now that your server will only contact Let’s Encrypt when a cert is eligible for renewal. I’ll set my cron job to run daily, considering it begins asking Let’s Encrypt for an update on the 30th day and would require 60 failures until the 90 day cert expires if all 60 renewal requests failed. It seems the tutorial was right after all.

Thank you all for the great support and advice! I wish support was this good on some other packages I’m attempting to install.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.