If I relocate the letsencrypt directory to another location and symlink it, certbot stops working. Is there a way to accomplish this? I'd like to put the letsencrypt directory on shared storage.
I'm curious as to what you are trying to accomplish by moving the directory.
[not taking anything for granted]
And also a bit nervous about the implications of the www in the path.
[I sure hope you aren't making any of that directly visible via an Internet browser!]
What shows: ls -lah /usr/local/etc/lets* ls -lah /www/lets* ls -lah /www/letsencrypt/
The reasoning behind the move is that I'm trying to make the web server more modular. I'm using ansible to instantiate web servers - infrastructure as code. If I could move the directory to the shared storage, then that would be great because then the certbot config and current status would move along with the websites stored on that particular storage backend and their configuration files.
No, I have no plans to expose the directory to the public. The /www directory is just the top level (mount) for all the web stuff that the servers publish, like so...
It's a head-scratcher for me because it's the first and only time I've ever seen the presence of a soft link affect the operation of an application. Usually they are invisible to apps.
An alternative approach to trying to make certbot configuration portable is to make your certificate portable instead. If you use a secrets store such as Ansible Vault or Hashicorp Vault you can keep your certificate there and pull the cert files down when you do your deployment (and when they change). You would then perform certificate requests/renewals either as part of deployment or as an ongoing scheduled script - dns validation is good for this because then the current state of your http servers isn't important.
A useful aspect of this is that your secrets store always has your latest certificate, so deployment won't fail just because your certificate request failed and you are less likely to hit a request rate limit.
@webprofusion, thanks for the thoughts. It sounds like a rather different setup than mine, though, so that would take some looking into. I'll research it a bit.
@schoen, thanks, I tried that. Looks like that doesn't solve it and the lock file gets created again...
% ls -lah /letc/letsencrypt/
total 71
drwxr-xr-x 9 root wheel 14B May 27 09:05 .
drwxrwxr-x 11 root wheel 12B May 26 19:06 ..
-rw------- 1 root wheel 0B May 27 09:05 .certbot.lock
-rw-r--r-- 1 root wheel 64B Jul 8 2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r-- 1 root wheel 64B Jul 8 2020 .updated-ssl-dhparams-pem-digest.txt
drwx------ 4 root wheel 4B Jul 8 2020 accounts
drwx------ 20 root wheel 20B Mar 16 17:51 archive
drwxr-xr-x 2 root wheel 187B May 17 10:15 csr
drwx------ 2 root wheel 187B May 17 10:15 keys
drwx------ 20 root wheel 21B Mar 16 17:51 live
-rw-r--r-- 1 root wheel 721B Jul 8 2020 options-ssl-nginx.conf
drwxr-xr-x 2 root wheel 19B May 17 10:15 renewal
drwxr-xr-x 5 root wheel 5B Jul 8 2020 renewal-hooks
-rw-r--r-- 1 root wheel 424B Jul 8 2020 ssl-dhparams.pem
% sudo rm /usr/local/etc/letsencrypt/.certbot.lock
% sudo certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
The following error was encountered:
[Errno 45] Operation not supported
Either run as root, or set --config-dir, --work-dir, and --logs-dir to writeable paths.
% ls -lah /usr/local/etc/letsencrypt/
total 71
drwxr-xr-x 9 root wheel 14B May 27 09:07 .
drwxrwxr-x 11 root wheel 12B May 26 19:06 ..
-rw------- 1 root wheel 0B May 27 09:07 .certbot.lock
-rw-r--r-- 1 root wheel 64B Jul 8 2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r-- 1 root wheel 64B Jul 8 2020 .updated-ssl-dhparams-pem-digest.txt
drwx------ 4 root wheel 4B Jul 8 2020 accounts
drwx------ 20 root wheel 20B Mar 16 17:51 archive
drwxr-xr-x 2 root wheel 187B May 17 10:15 csr
drwx------ 2 root wheel 187B May 17 10:15 keys
drwx------ 20 root wheel 21B Mar 16 17:51 live
-rw-r--r-- 1 root wheel 721B Jul 8 2020 options-ssl-nginx.conf
drwxr-xr-x 2 root wheel 19B May 17 10:15 renewal
drwxr-xr-x 5 root wheel 5B Jul 8 2020 renewal-hooks
-rw-r--r-- 1 root wheel 424B Jul 8 2020 ssl-dhparams.pem
Another alternative to making the certbot config portable is to use a different client that's more conducive to being portable like acme.sh. It has a number of parameters related to config locations. Though I haven't messed around with them myself.
--accountconf <file> Specifies a customized account config file.
--home <directory> Specifies the home dir for acme.sh.
--cert-home <directory> Specifies the home dir to save all the certs, only valid for '--install' command.
--config-home <directory> Specifies the home dir to save all the configurations.
On our machines that run Certbot, we have been successfully using a symlinked directory for about 5 years, if not longer. The earliest current one still in production dates back to 2017, but I've been doing this since first using LetsEncrypt.
Some questions:
1- What is your operating system?
2- What is your disk formatted as?
3- How did you create the symlink?
In case anyone is curious why, our cloud servers are built from multiple images that are loaded onto different partitions:
Image 1 - Core Linux and Support Applications
Image 2 - Our Applications
Image 3 - LetsEncrypt and some other persistent configs
This strategy lets us switch out the linux version and upgrade/downgrade/revert instantly.
Thanks everyone for the helpful replies. I've been busy this week, so have to circle back to this soon to do some more testing. In the meantime, @jvanasco...
Sorry I missed that. I use the same command. We're ubuntu with ext3 and ext4 - I should have mentioned that.
If I have time, I'll try to test this on a NFS share and ZFS. I definitely don't have time to test FreeBSD.
IMHO, this is likely due to :
NFS
ZFS
FreeBSD
Just throwing this out there - have you tried setting the permissions to 777, just to rule out some permission issue in a nested directory? In my experience, the groups and users tend to get messed up when dealing with network shares and sometimes there is a deeply nested directory with the wrong permissions. I doubt this is the case here, but I wanted to mention it.
I think that explains it because lockf generally doesn't work on NFS mounts. Certbot is insisting on taking an OS-level lock on its lockfile, but NFS implementations are, I think, unable to enforce that lock (whether because of an oversight or because the semantics of a network filesystem would inherently not allow it to be enforced, I'm not sure). Therefore the system call probably returns an error saying that the lock operation is unsupported.
If you find your desired approach important, we could ask the Certbot developers to consider allowing the lockfile to be located in a different place from the rest of the work directory, but I think they might be reluctant because ideally you should be able to have separate lockfiles for distinct work directories.
That would also explain why my installation works. It's not an NFS mount, but a single virtual device/image. Only one server ever writes to it; the others just read replicas.
I tried creating the link before installing certbot, but that didn't make a difference. Note that the installation routine doesn't put anything into that dir anyway. I'm not trying to move the executables, just the config and status.
I didn't try acme.sh because I want to keep the setup as simple as possible and don't want to change horses at this time, but will keep it on the radar and do some testing at some point.
The lockf issue on NFS sounds like that is the real issue. I wonder if it would work over ISCSI instead of NFS? I think I'll give that a try and reply back here.
I found an elegant solution to this problem. After some thought, I realized I don't need to run certbot on the front-end server. I resolved this issue by simply moving certbot, and its cron job to the storage server. The symlink now works, because it's local, so the letsencrypt config is now stored with the www config and files. In order to keep things elegant and let certbot find the config, I installed and configured the nginx package on the storage server, but did not enable the service there.