Certbot Hardlink Mode


#1

A third-party app I am trying to use has some unfortunate containment system that requires me to bind-mount /etc/letsencrypt/live/domain.com, and that means symlinks become broken. To make this work, I had to replace the symlinks in /etc/letsencrypt/live/domain.com/ with hardlinks. I would like to request a hardlink mode where it does this for me.


#2

Can you bind mount all of /etc/letsencrypt/, or use a deploy hook to copy the files to another directory?

Certbot won’t be able to renew the certificate unless you change the symlinks back.


#3

Can you bind mount all of /etc/letsencrypt/

That would be overkill, and the symlinks would still be broken because of the way the container works. The host manages the certificates and renewal, while the container just needs access to them.

or use a deploy hook to copy the files to another directory?

I suppose I could do that. I would have to look into the docs on this though, because I’ve never done it before.

Certbot won’t be able to renew the certificate unless you change the symlinks back.

Why not?


#4

Why would they be broken? They’re relative, so they should work.

It’s very simple… and also complicated.

Depending on your OS and Certbot version and how it’s automated, you might want to put a simple script that runs cp in /etc/letsencrypt/renewal-hooks/deploy, or add “--deploy-hook /path/to/script” to “certbot renew”, or something else.

It expects them to be symlinks.


#5

The symlink should always remain the same (a symlink).
It merely gets updated to point to the latest file.


#6

In Unix, symlinks and hardlinks have different implementations: when you have hardlinks, there is no notion that one is the “original” because both are pointers to the same file. When you have softlinks, they indicate a specific path to the target (which can be examined by software).

schoen@host:~$ mkdir example
schoen@host:~$ cd example
schoen@host:~/example$ echo foo > original
schoen@host:~/example$ ln -s original symlink
schoen@host:~/example$ ln original hardlink
schoen@host:~/example$ ls -l
total 8
-rw-rw-r-- 2 schoen schoen 4 Dec 31 16:39 hardlink
-rw-rw-r-- 2 schoen schoen 4 Dec 31 16:39 original
lrwxrwxrwx 1 schoen schoen 8 Dec 31 16:39 symlink -> original
schoen@demorgan:~/example$ ls -li
total 8
1582713 -rw-rw-r-- 2 schoen schoen 4 Dec 31 16:39 hardlink
1582713 -rw-rw-r-- 2 schoen schoen 4 Dec 31 16:39 original
1583328 lrwxrwxrwx 1 schoen schoen 8 Dec 31 16:39 symlink -> original
schoen@host:~/example$

Here you can see that the two hardlinks point to the same inode and are indistinguishable. In fact, the new hardlink is just as valid as the original name even if the original name is removed:

schoen@host:~/example$ rm original 
schoen@host:~/example$ cat symlink
cat: symlink: No such file or directory
schoen@host:~/example$ cat hardlink
foo
schoen@host:~/example$ 

In this sense, hardlinks convey less information than softlinks. When we designed the file storage for Certbot, we deliberately took advantage of the way that information could be stored in softlinks in order to maintain and track versions of related files, while allowing a consistent path that web server configurations could point to.

This choice might not be ideal in various ways, but it has some benefits and would be difficult to change in the context of the current Certbot design. (Another options that we considered, which in retrospect might have had other benefits, was keeping all of the historic information in a database file such as a sqlite database, and only presenting the most recent version of each file to the user in /etc/letsencrypt. One reason that we went with the approach we did was that we thought that sysadmins might be very interested in the certificate history and in the ability to roll back to an earlier version of a certificate. In practice, very few people have taken advantage of that possibility with any frequency.)


#7

Because the way the container virtualizes the environment doesn’t support symlinks. It’s really weird.

@schoen said it better.

I suppose I will do the deploy hooks. It seems to be the better of options, since I have something else I have to update by hand anyway. Thanks for the suggestions and explanations everyone.