The reasoning behind the move is that I'm trying to make the web server more modular. I'm using ansible to instantiate web servers - infrastructure as code. If I could move the directory to the shared storage, then that would be great because then the certbot config and current status would move along with the websites stored on that particular storage backend and their configuration files.
No, I have no plans to expose the directory to the public. The /www directory is just the top level (mount) for all the web stuff that the servers publish, like so...
It's a head-scratcher for me because it's the first and only time I've ever seen the presence of a soft link affect the operation of an application. Usually they are invisible to apps.
An alternative approach to trying to make certbot configuration portable is to make your certificate portable instead. If you use a secrets store such as Ansible Vault or Hashicorp Vault you can keep your certificate there and pull the cert files down when you do your deployment (and when they change). You would then perform certificate requests/renewals either as part of deployment or as an ongoing scheduled script - dns validation is good for this because then the current state of your http servers isn't important.
A useful aspect of this is that your secrets store always has your latest certificate, so deployment won't fail just because your certificate request failed and you are less likely to hit a request rate limit.
Another alternative to making the certbot config portable is to use a different client that's more conducive to being portable like acme.sh. It has a number of parameters related to config locations. Though I haven't messed around with them myself.
--accountconf <file> Specifies a customized account config file.
--home <directory> Specifies the home dir for acme.sh.
--cert-home <directory> Specifies the home dir to save all the certs, only valid for '--install' command.
--config-home <directory> Specifies the home dir to save all the configurations.
On our machines that run Certbot, we have been successfully using a symlinked directory for about 5 years, if not longer. The earliest current one still in production dates back to 2017, but I've been doing this since first using LetsEncrypt.
1- What is your operating system?
2- What is your disk formatted as?
3- How did you create the symlink?
In case anyone is curious why, our cloud servers are built from multiple images that are loaded onto different partitions:
Image 1 - Core Linux and Support Applications
Image 2 - Our Applications
Image 3 - LetsEncrypt and some other persistent configs
This strategy lets us switch out the linux version and upgrade/downgrade/revert instantly.
Sorry I missed that. I use the same command. We're ubuntu with ext3 and ext4 - I should have mentioned that.
If I have time, I'll try to test this on a NFS share and ZFS. I definitely don't have time to test FreeBSD.
IMHO, this is likely due to :
Just throwing this out there - have you tried setting the permissions to 777, just to rule out some permission issue in a nested directory? In my experience, the groups and users tend to get messed up when dealing with network shares and sometimes there is a deeply nested directory with the wrong permissions. I doubt this is the case here, but I wanted to mention it.
I think that explains it because lockf generally doesn't work on NFS mounts. Certbot is insisting on taking an OS-level lock on its lockfile, but NFS implementations are, I think, unable to enforce that lock (whether because of an oversight or because the semantics of a network filesystem would inherently not allow it to be enforced, I'm not sure). Therefore the system call probably returns an error saying that the lock operation is unsupported.
If you find your desired approach important, we could ask the Certbot developers to consider allowing the lockfile to be located in a different place from the rest of the work directory, but I think they might be reluctant because ideally you should be able to have separate lockfiles for distinct work directories.
I tried creating the link before installing certbot, but that didn't make a difference. Note that the installation routine doesn't put anything into that dir anyway. I'm not trying to move the executables, just the config and status.
I didn't try acme.sh because I want to keep the setup as simple as possible and don't want to change horses at this time, but will keep it on the radar and do some testing at some point.
The lockf issue on NFS sounds like that is the real issue. I wonder if it would work over ISCSI instead of NFS? I think I'll give that a try and reply back here.
I found an elegant solution to this problem. After some thought, I realized I don't need to run certbot on the front-end server. I resolved this issue by simply moving certbot, and its cron job to the storage server. The symlink now works, because it's local, so the letsencrypt config is now stored with the www config and files. In order to keep things elegant and let certbot find the config, I installed and configured the nginx package on the storage server, but did not enable the service there.