Relocate Letsencrypt Directory

If I relocate the letsencrypt directory to another location and symlink it, certbot stops working. Is there a way to accomplish this? I'd like to put the letsencrypt directory on shared storage.

Certbot works fine:

% ls -lah /www/letsencrypt
total 72
drwxr-xr-x   9 root  wheel    15B May 26 19:07 .
drwxrwxr-x  11 root  wheel    12B May 26 19:06 ..
-rw-------   1 root  wheel     0B May 26 19:07 .certbot.lock
-rw-------   1 root  wheel     0B May 21 10:10 .certbot.lock.bak
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-ssl-dhparams-pem-digest.txt
drwx------   4 root  wheel     4B Jul  8  2020 accounts
drwx------  20 root  wheel    20B Mar 16 17:51 archive
drwxr-xr-x   2 root  wheel   187B May 17 10:15 csr
drwx------   2 root  wheel   187B May 17 10:15 keys
drwx------  20 root  wheel    21B Mar 16 17:51 live
-rw-r--r--   1 root  wheel   721B Jul  8  2020 options-ssl-nginx.conf
drwxr-xr-x   2 root  wheel    19B May 17 10:15 renewal
drwxr-xr-x   5 root  wheel     5B Jul  8  2020 renewal-hooks
-rw-r--r--   1 root  wheel   424B Jul  8  2020 ssl-dhparams.pem

% sudo certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

But then moving and linking the dir causes error:

% sudo cp -pr /usr/local/etc/letsencrypt /www/

% sudo mv /usr/local/etc/letsencrypt /usr/local/etc/letsencrypt.bak

% sudo ln -s /www/letsencrypt /usr/local/etc/letsencrypt

% ls -lah /usr/local/etc/letsencrypt
lrwxr-xr-x  1 root  wheel    16B May 26 19:17 /usr/local/etc/letsencrypt -> /www/letsencrypt

 % ls -lah /usr/local/etc/letsencrypt/
total 72
drwxr-xr-x   9 root  wheel    15B May 26 19:07 .
drwxrwxr-x  11 root  wheel    12B May 26 19:06 ..
-rw-------   1 root  wheel     0B May 26 19:07 .certbot.lock
-rw-------   1 root  wheel     0B May 21 10:10 .certbot.lock.bak
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-ssl-dhparams-pem-digest.txt
drwx------   4 root  wheel     4B Jul  8  2020 accounts
drwx------  20 root  wheel    20B Mar 16 17:51 archive
drwxr-xr-x   2 root  wheel   187B May 17 10:15 csr
drwx------   2 root  wheel   187B May 17 10:15 keys
drwx------  20 root  wheel    21B Mar 16 17:51 live
-rw-r--r--   1 root  wheel   721B Jul  8  2020 options-ssl-nginx.conf
drwxr-xr-x   2 root  wheel    19B May 17 10:15 renewal
drwxr-xr-x   5 root  wheel     5B Jul  8  2020 renewal-hooks
-rw-r--r--   1 root  wheel   424B Jul  8  2020 ssl-dhparams.pem

% sudo certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
The following error was encountered:
[Errno 45] Operation not supported
Either run as root, or set --config-dir, --work-dir, and --logs-dir to writeable paths.
2 Likes

I'm curious as to what you are trying to accomplish by moving the directory.
[not taking anything for granted]

And also a bit nervous about the implications of the www in the path.
[I sure hope you aren't making any of that directly visible via an Internet browser!]

What shows:
ls -lah /usr/local/etc/lets*
ls -lah /www/lets*
ls -lah /www/letsencrypt/

2 Likes

The reasoning behind the move is that I'm trying to make the web server more modular. I'm using ansible to instantiate web servers - infrastructure as code. If I could move the directory to the shared storage, then that would be great because then the certbot config and current status would move along with the websites stored on that particular storage backend and their configuration files.

No, I have no plans to expose the directory to the public. The /www directory is just the top level (mount) for all the web stuff that the servers publish, like so...

It's a head-scratcher for me because it's the first and only time I've ever seen the presence of a soft link affect the operation of an application. Usually they are invisible to apps.

Note: the storage mount is read-write.

% tree -L 1 /www/
/www/
├── conf
├── letsencrypt
├── nginx -> /usr/local/www/nginx-dist
├── php-fpm
├── tls
└── vhosts

Here are the requested outputs...

% ls -lah /usr/local/etc/lets*
lrwxr-xr-x  1 root  wheel    16B May 26 19:17 /usr/local/etc/letsencrypt -> /www/letsencrypt


% ls -lah /www/lets*
/www/letsencrypt:
total 72
drwxr-xr-x   9 root  wheel    15B May 26 19:07 .
drwxrwxr-x  11 root  wheel    12B May 26 19:06 ..
-rw-------   1 root  wheel     0B May 26 19:07 .certbot.lock
-rw-------   1 root  wheel     0B May 21 10:10 .certbot.lock.bak
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-ssl-dhparams-pem-digest.txt
drwx------   4 root  wheel     4B Jul  8  2020 accounts
drwx------  20 root  wheel    20B Mar 16 17:51 archive
drwxr-xr-x   2 root  wheel   187B May 17 10:15 csr
drwx------   2 root  wheel   187B May 17 10:15 keys
drwx------  20 root  wheel    21B Mar 16 17:51 live
-rw-r--r--   1 root  wheel   721B Jul  8  2020 options-ssl-nginx.conf
drwxr-xr-x   2 root  wheel    19B May 17 10:15 renewal
drwxr-xr-x   5 root  wheel     5B Jul  8  2020 renewal-hooks
-rw-r--r--   1 root  wheel   424B Jul  8  2020 ssl-dhparams.pem


% ls -lah /www/letsencrypt/
total 72
drwxr-xr-x   9 root  wheel    15B May 26 19:07 .
drwxrwxr-x  11 root  wheel    12B May 26 19:06 ..
-rw-------   1 root  wheel     0B May 26 19:07 .certbot.lock
-rw-------   1 root  wheel     0B May 21 10:10 .certbot.lock.bak
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-ssl-dhparams-pem-digest.txt
drwx------   4 root  wheel     4B Jul  8  2020 accounts
drwx------  20 root  wheel    20B Mar 16 17:51 archive
drwxr-xr-x   2 root  wheel   187B May 17 10:15 csr
drwx------   2 root  wheel   187B May 17 10:15 keys
drwx------  20 root  wheel    21B Mar 16 17:51 live
-rw-r--r--   1 root  wheel   721B Jul  8  2020 options-ssl-nginx.conf
drwxr-xr-x   2 root  wheel    19B May 17 10:15 renewal
drwxr-xr-x   5 root  wheel     5B Jul  8  2020 renewal-hooks
-rw-r--r--   1 root  wheel   424B Jul  8  2020 ssl-dhparams.pem
2 Likes

An alternative approach to trying to make certbot configuration portable is to make your certificate portable instead. If you use a secrets store such as Ansible Vault or Hashicorp Vault you can keep your certificate there and pull the cert files down when you do your deployment (and when they change). You would then perform certificate requests/renewals either as part of deployment or as an ongoing scheduled script - dns validation is good for this because then the current state of your http servers isn't important.

A useful aspect of this is that your secrets store always has your latest certificate, so deployment won't fail just because your certificate request failed and you are less likely to hit a request rate limit.

3 Likes

It looks like this has to do with Certbot's lockfile mechanism (I think the errno 45 is from the lockf call in Certbot). Are you on FreeBSD here?

If you're sure you have no other Certbot processes running at all, including from cron, can you try removing .certbot.lock and then try Certbot again?

3 Likes

@webprofusion, thanks for the thoughts. It sounds like a rather different setup than mine, though, so that would take some looking into. I'll research it a bit.

@schoen, thanks, I tried that. Looks like that doesn't solve it and the lock file gets created again...

% ls -lah /letc/letsencrypt/
total 71
drwxr-xr-x   9 root  wheel    14B May 27 09:05 .
drwxrwxr-x  11 root  wheel    12B May 26 19:06 ..
-rw-------   1 root  wheel     0B May 27 09:05 .certbot.lock
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-ssl-dhparams-pem-digest.txt
drwx------   4 root  wheel     4B Jul  8  2020 accounts
drwx------  20 root  wheel    20B Mar 16 17:51 archive
drwxr-xr-x   2 root  wheel   187B May 17 10:15 csr
drwx------   2 root  wheel   187B May 17 10:15 keys
drwx------  20 root  wheel    21B Mar 16 17:51 live
-rw-r--r--   1 root  wheel   721B Jul  8  2020 options-ssl-nginx.conf
drwxr-xr-x   2 root  wheel    19B May 17 10:15 renewal
drwxr-xr-x   5 root  wheel     5B Jul  8  2020 renewal-hooks
-rw-r--r--   1 root  wheel   424B Jul  8  2020 ssl-dhparams.pem

% sudo rm /usr/local/etc/letsencrypt/.certbot.lock

% sudo certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
The following error was encountered:
[Errno 45] Operation not supported
Either run as root, or set --config-dir, --work-dir, and --logs-dir to writeable paths.

% ls -lah /usr/local/etc/letsencrypt/
total 71
drwxr-xr-x   9 root  wheel    14B May 27 09:07 .
drwxrwxr-x  11 root  wheel    12B May 26 19:06 ..
-rw-------   1 root  wheel     0B May 27 09:07 .certbot.lock
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-options-ssl-nginx-conf-digest.txt
-rw-r--r--   1 root  wheel    64B Jul  8  2020 .updated-ssl-dhparams-pem-digest.txt
drwx------   4 root  wheel     4B Jul  8  2020 accounts
drwx------  20 root  wheel    20B Mar 16 17:51 archive
drwxr-xr-x   2 root  wheel   187B May 17 10:15 csr
drwx------   2 root  wheel   187B May 17 10:15 keys
drwx------  20 root  wheel    21B Mar 16 17:51 live
-rw-r--r--   1 root  wheel   721B Jul  8  2020 options-ssl-nginx.conf
drwxr-xr-x   2 root  wheel    19B May 17 10:15 renewal
drwxr-xr-x   5 root  wheel     5B Jul  8  2020 renewal-hooks
-rw-r--r--   1 root  wheel   424B Jul  8  2020 ssl-dhparams.pem
3 Likes

Another alternative to making the certbot config portable is to use a different client that's more conducive to being portable like acme.sh. It has a number of parameters related to config locations. Though I haven't messed around with them myself.

  --accountconf <file>              Specifies a customized account config file.
  --home <directory>                Specifies the home dir for acme.sh.
  --cert-home <directory>           Specifies the home dir to save all the certs, only valid for '--install' command.
  --config-home <directory>         Specifies the home dir to save all the configurations.
3 Likes

On our machines that run Certbot, we have been successfully using a symlinked directory for about 5 years, if not longer. The earliest current one still in production dates back to 2017, but I've been doing this since first using LetsEncrypt.

Some questions:

1- What is your operating system?
2- What is your disk formatted as?
3- How did you create the symlink?

In case anyone is curious why, our cloud servers are built from multiple images that are loaded onto different partitions:

  • Image 1 - Core Linux and Support Applications
  • Image 2 - Our Applications
  • Image 3 - LetsEncrypt and some other persistent configs

This strategy lets us switch out the linux version and upgrade/downgrade/revert instantly.

4 Likes

Thanks everyone for the helpful replies. I've been busy this week, so have to circle back to this soon to do some more testing. In the meantime, @jvanasco...

  1. FreeBSD FreeBSD 12.2-RELEASE-p6
  2. The /www is an NFS mount backed by ZFS.
  3. You can see the creation of the link in the OP.
3 Likes

Sorry I missed that. I use the same command. We're ubuntu with ext3 and ext4 - I should have mentioned that.

If I have time, I'll try to test this on a NFS share and ZFS. I definitely don't have time to test FreeBSD.

IMHO, this is likely due to :

  • NFS
  • ZFS
  • FreeBSD

Just throwing this out there - have you tried setting the permissions to 777, just to rule out some permission issue in a nested directory? In my experience, the groups and users tend to get messed up when dealing with network shares and sometimes there is a deeply nested directory with the wrong permissions. I doubt this is the case here, but I wanted to mention it.

2 Likes

Can you try creating the linked directory before installing certbot?
[If it does install to the shared location then it should also work from there]

3 Likes

I think that explains it because lockf generally doesn't work on NFS mounts. Certbot is insisting on taking an OS-level lock on its lockfile, but NFS implementations are, I think, unable to enforce that lock (whether because of an oversight or because the semantics of a network filesystem would inherently not allow it to be enforced, I'm not sure). Therefore the system call probably returns an error saying that the lock operation is unsupported.

If you find your desired approach important, we could ask the Certbot developers to consider allowing the lockfile to be located in a different place from the rest of the work directory, but I think they might be reluctant because ideally you should be able to have separate lockfiles for distinct work directories.

4 Likes

That would also explain why my installation works. It's not an NFS mount, but a single virtual device/image. Only one server ever writes to it; the others just read replicas.

3 Likes

Finally circling back to this now.

I tried creating the link before installing certbot, but that didn't make a difference. Note that the installation routine doesn't put anything into that dir anyway. I'm not trying to move the executables, just the config and status.

I didn't try acme.sh because I want to keep the setup as simple as possible and don't want to change horses at this time, but will keep it on the radar and do some testing at some point.

The lockf issue on NFS sounds like that is the real issue. I wonder if it would work over ISCSI instead of NFS? I think I'll give that a try and reply back here.

Thanks everyone for your helpful replies!

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

I found an elegant solution to this problem. After some thought, I realized I don't need to run certbot on the front-end server. I resolved this issue by simply moving certbot, and its cron job to the storage server. The symlink now works, because it's local, so the letsencrypt config is now stored with the www config and files. In order to keep things elegant and let certbot find the config, I installed and configured the nginx package on the storage server, but did not enable the service there.

5 Likes

Oh, and I needed to add a hook to restart nginx on the remote, front-end web server(s).

2 Likes