Symlinks disappear - Windows 10, WSL, Docker

Help
1

The symlinks in the /live/[domain] name keep disappearing within 1 minute.

I'm running a Docker container for self-hosted bitwarden. The certificate is created and works fine. However, the symlinks in the /live/[domain] folder pointing to the files in the /archive/[domain] folder keep disappearing, resulting in simple 0-byte files in the /live/[domain] folder.

I have recreated the symlink files in the WSL console and from Windows with the same result.

I am able to create symlinks in the parent folder folder - /live (i.e. not in the [domain] subfolder) and the symlinks remain stable.

My domain is: bw,davekiernan.com

I ran this command: dir

It produced this output:
right after creating the symlinks
07/05/2022 08:02 PM cert.pem [....\archive\bw.davekiernan.com\cert3.pem]
07/05/2022 08:02 PM chain.pem [....\archive\bw.davekiernan.com\chain3.pem]
12/05/2021 11:18 AM 424 dhparam.pem
07/05/2022 08:02 PM fullchain.pem [....\archive\bw.davekiernan.com\fullchain3.pem]
07/05/2022 08:02 PM privkey.pem [....\archive\bw.davekiernan.com\privkey3.pem]

one minute after getting the output above, the same dir produces:
07/05/2022 08:02 PM 0 cert.pem
07/05/2022 08:02 PM 0 chain.pem
12/05/2021 11:18 AM 424 dhparam.pem
07/05/2022 08:02 PM 0 fullchain.pem
07/05/2022 08:02 PM 0 privkey.pem

Notice the timestamps are the same, but the files are no longer symlinks.

My web server is (include version): nginx

The operating system my web server runs on is (include version): Windows 10, Docker

My hosting provider, if applicable, is: self hosted bitwarden

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): n/a

2

Hi @2112.snowdog, and welcome to the LE community forum :slight_smile:

Please show the output of:
ls -l /etc/letsencrypt/live/
[or wherever the live folder is at]

And the relevant Docker settings for that location.

3 Likes
3

I'll get the ls -l output later today or tomorrow.
As you can (kind of) see from the DIR command, the files are symlinks and within a minute or so, the symlink is not shown even though the file timestamps remain unchanged.

What would you consider "relevant Docker settings for the location". The Docker settings on the /live/[domain] folder where the symlinks are disappearing are (AFAICT) the same as the /live folder where I've created symlinks to the same target that have not disappeared.
Is there something in particular in the Docker settings I should be looking for?

4

The folders being used and their access.

What do you mean?:

2 Likes
5

Is your data store on a Windows volume or only in Linux (e.g. part of your WSL distro)? I don't think symlinks within linux back to a windows volume work in WSL.

[edit: it could also just be the windows integration e.g. DIR run against a WSL volume doesn't see the symlink but it's still there in linux]

3 Likes
6

@webprofusion - See below, the folders are all in the same subfolder, and creating symlinks in the /live/[domain] folder disappear, but using the same process to create symlink in the /live folder does not result in the symlink disappearing. The entire folder structure is a Windows volume that I can access in WSL via /mnt/d/..../letsencrypt/live

Here I have manually created the symlinks in a WSL shell session, then immediately did an ls -l. Within 1 minute, the symlinks are gone as shown by an ls -l 1 minute later.

Create symlinks - 08:54
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ln -s ../../archive/bw.davekiernan.com/cert3.pem live/bw.davekiernan.com/cert.pem
ln: failed to create symbolic link 'live/bw.davekiernan.com/cert.pem': File exists
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ln -s ../../archive/bw.davekiernan.com/cert3.pem live/bw.davekiernan.com/cert.pem
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ln -s ../../archive/bw.davekiernan.com/chain3.pem live/bw.davekiernan.com/chain.pem
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ln -s ../../archive/bw.davekiernan.com/fullchain3.pem live/bw.davekiernan.com/fullchain.pem
-s ../../archive/bw.davekiernan.com/privkey3.pem live/bw.davekiernan.com/privkey.pembitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ln -s ../../archive/bw.davekiernan.com/privkey3.pem live/bw.davekiernan.com/privkey.pem
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ls -l live/bw.davekiernan.com
total 4
-rw-r--r-- 1 nobody nogroup 692 Dec 5 2021 README
lrwxrwxrwx 1 bitwarden bitwarden 42 Jul 12 08:54 cert.pem -> ../../archive/bw.davekiernan.com/cert3.pem
lrwxrwxrwx 1 bitwarden bitwarden 43 Jul 12 08:54 chain.pem -> ../../archive/bw.davekiernan.com/chain3.pem
-rw-r--r-- 1 nobody nogroup 424 Dec 5 2021 dhparam.pem
lrwxrwxrwx 1 bitwarden bitwarden 47 Jul 12 08:54 fullchain.pem -> ../../archive/bw.davekiernan.com/fullchain3.pem
lrwxrwxrwx 1 bitwarden bitwarden 45 Jul 12 08:54 privkey.pem -> ../../archive/bw.davekiernan.com/privkey3.pem
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$

1 minute later - 08:55
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ls -l live/bw.davekiernan.com
total 4
-rw-r--r-- 1 nobody nogroup 692 Dec 5 2021 README
-rwxrwxrwx 1 bitwarden bitwarden 0 Jul 12 08:54 cert.pem
-rwxrwxrwx 1 bitwarden bitwarden 0 Jul 12 08:54 chain.pem
-rw-r--r-- 1 nobody nogroup 424 Dec 5 2021 dhparam.pem
-rwxrwxrwx 1 bitwarden bitwarden 0 Jul 12 08:54 fullchain.pem
-rwxrwxrwx 1 bitwarden bitwarden 0 Jul 12 08:54 privkey.pem
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$

I created symlinks manually in letsencrypt/live folder (not in the [domain] subfolder) on 7/2/22. Those symlinks are still "intact".

bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$ sudo ls -l live
total 4
-rw-r--r-- 1 nobody nogroup 740 Dec 5 2021 README
drwxr-xr-x 1 nobody nogroup 4096 Jul 12 08:54 bw.davekiernan.com
lrwxrwxrwx 1 nobody nogroup 42 Jul 5 20:00 cert.pem -> ../../archive/bw.davekiernan.com/cert3.pem
lrwxrwxrwx 1 nobody nogroup 43 Jul 5 20:00 chain.pem -> ../../archive/bw.davekiernan.com/chain3.pem
bitwarden@FILESERVER:/mnt/d/docker/bitwarden/bwdata/letsencrypt$

7

Sorry, I don't know. My guess would be that because the underlying symlink is created in the NTFS filesystem (not a native linux filesystem) that it gets lost once the in-memory filesystem proxy flushes it's cache. It could be related to permissions as the failing links seems to be a different user. Best check out the open issues in WSL as this is not a Let's Encrypt or certbot bug, it's a WSL or Windows behavior. Issues · microsoft/WSL · GitHub

4 Likes
8

By way of comparison, I have another system running a very similar configuration - also a bitwarden docker container with lets encrypt. On this other system, I am not having issues with the symlinks.

You've mentioned several times "not a linux filesystem". Is this supposed to work on Windows running WSL? Again, I come back to: I can create symlinks in other folders in the same folder tree within the docker container and they do not disappear. It's only the symlinks in the /live/[domain] folder.

What are my options at this point?

Here are the resource settings in the docker config related to filesystem:

        "Mounts": [
            {
                "Type": "bind",
                "Source": "D:\\docker\\bitwarden\\bwdata\\nginx",
                "Destination": "/etc/bitwarden/nginx",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "D:\\docker\\bitwarden\\bwdata\\letsencrypt",
                "Destination": "/etc/letsencrypt",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "D:\\docker\\bitwarden\\bwdata\\logs\\nginx",
                "Destination": "/var/log/nginx",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "D:\\docker\\bitwarden\\bwdata\\ssl",
                "Destination": "/etc/ssl",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
9

I figured out the issue.

The symlinks are indeed on a Windows disk - a Windows 10 system with WSL2 installed (ubuntu 20.04 distro), running a docker container for Bitwarden.

Nothing unusual there - as mentioned, I have another similar system that has been working fine.

The difference: the non-working system is running GOOGLE DRIVE to sync the bitwarden folder to Google Drive, including the letsencrypt folder.

I stopped syncing the letencrypt folder and symlinks are working just fine...so far.

I think it's clear the google sync process is messing with symlinks. If something changes in the next 24 hours, I'll report back. If not - we can call this mystery: solved.

3 Likes
10

Cool, I don't use Google Drive anymore but Onedrive has a space saving feature that replaces local files with a stub and often that kind of stuff uses ntfs alternate data streams to attach data, perhaps it's something like that getting mangled by Google Drive.

3 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.