Apache SSLCertificateFile error: Does not exist or is empty

I renewed three certs today. Since all those are wildcards, I was using a manual method:
certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns -d *.seeotter.tv -d seeotter.tv
I also yum updated certbot to 0.34.2

Everything went fine, although waiting for gandi.net to propagate DNS really takes a lot of time.

However, we have a couple of servers, and I had to xfr those certs to the other box. Previously (in March), I used tar to get /etc/letsencrypt folder, and scp it to the other box. This time, I did the same. On the other box, I untared it, swapped /etc/letsencrypt to the new one. Once I restarted httpd (systemctl), httpd couldn’t start complaining about a missing file (privkey.pem, as it was listed as the first one in ssl.conf). The file was there. I compared directory structure with ‘ls’ and ‘namei -mo’, but everything was exactly the same. I tried to disable SELinux (CentOS 7), but that didn’t help either. I tried to find something in the log files, but that also didn’t provide any clues.

Finaly, I manually copied all new certs (privkey, fullchain, chain, cert) for all three domains to the old /etc/letsencrypt/archive directory and manually regenerated symlinks in live folder. That fixed the problem.

I have just did another test, where in the old folder structure, I tried to use the ‘original’ (that came from tar) privkey.pem and the copied one. With the original, httpd complained that file was missing.

I cannot tell what is the difference between the file that came from tar, and the one I copied. Of course, since I copied it, whatever permissions were given to this new file were based on this box and the local root user. Although, as far as I can tell, those files are identical (content, permissions, etc. wise).

Again, this procedure worked in March. Something got changed afterwards, not sure whether due to yum updates, or certbot changes.

Thank you, Jacek

Hi @_jacek

that's typically a right problem. The http process can't work with the file -> missing file.

Not sure whether I understand your statement. The file is there, somehow http cannot access it. Since the file is virtually identical to its own copy, the issue is with some permissions that were either setup by certbot, or the original server (I assume the cp regenerated all permissions). However, I am not that good at Linux to be able to tell what permissions those are (based on ls, namei, SELinux, everything looks the same).

Apache is usually started as root and later changes to a less privileged user (e.g. www-data or such). Certificates are read as root user. You can check that the content of a certificate file are ok using the ‘openssl’ command:

> openssl x509 -text -in path-to-file

I did check that, and openssl show no problems at all.
Again, file moved from the original box doesn’t work, file copied from that original does.
Plus, the same procedure worked in March (with older certbot).
So, if possible I would like to find out what is causing this problem, as the current work-around is a bit painful.

Maybe the key does not match and therefor the certificate cannot be loaded successfully when apache starts?

Again, two identical files. The original one came as tarred version, the duplicate is ‘cp’ version of the original. No difference in those two files, as far as I can tell. One works, the other doesn’t.

1 Like

That would be a first in digital history. Chances are higher that you are overlooking something.

1 Like

I know that I am missing something; therefore I posted it here.
I tried it once more, results as follow:
diff cert6.pem-tar cert6.pem-cp // no difference
ls -la cert6*
-rw-r–r--. 1 root root 1923 Jun 17 14:39 cert6.pem-cp
-rw-r–r--. 1 root root 1923 Jun 17 12:28 cert6.pem-tar
same
openssl x509 -text -in cert6.pem-tar
openssl x509 -text -in cert6.pem-cp
same
namei -mo cert6.pem-cp
f: cert6.pem
-rw-r–r-- root root cert6.pem-cp
namei -mo cert6.pem-tar
f: cert6.pem-tar
-rw-r–r-- root root cert6.pem-tar
same
systemctl restart httpd // based on cert6.pem-cp - fine
systemctl restart httpd // based on cert6.pem-tar - error

1 Like

Today, I went back to check again at SELinux. Well, that was the problem. Yesterday, I was disabling SELinux; however, as that is a live server, I was rushing from one test to another, and had to have a secondary issue there, while testing SELinux (not properly moved all folders/symlinks, …). After disabling SELinux http started fine.

So, to reflect on the "that would be a first in digital history" comment, it is always like that, if the issue is above your paygrade. Whatever you were pointing to was fine. Those files were diff-identical, ls-identical, openssl-identical. Actually, those files were 'ls -la' identical, but not 'ls -Z' identical (-Z provides SELinux info for a given file).

In this post:

@JuergenAuer and @sahsanu endorse using tar to move certs from one server to another, as tar preserves all permissions, ownerships. This is exactly what I did. However, SELinux is recognizing such files as coming from a different server, and unless told to ignore that, will basically stop http from starting (using those files). Maybe it would be worthwhile to amend that post to mention potential issues with SELinux.

Thank you for helping me to resolve this issue and learn a bit about SELinux, Jacek

3 Likes

Selinux works on inodes, while the competing system, Apparmor works on file names.
To be candid, that’s about the sum of my knowledge on Selinux.
But maybe it could be useful. With the additional info that most people choosing Apparmor are doing so because Selinux is darn difficult to master.

2 Likes

Thanks for the summary of the resolution. Learned something as well.

1 Like

Fascinating! Thank you for letting us know about this issue—it will be helpful in case other people encounter this problem in the future.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.