SSL certificate does not exist or is empty: alternative root cause and solution

Server
1

hi there,

I was facing the same error as described in this post:

Whilst the error was similar, the root cause and solution is not. Since that thread is closed, I want to share an alternative possible root cause for this error, and what solved it for me. Just to avoid others having the same pain.

I'm running a Debian VM using VMWare on Windows 10. I have linked up two domains to that Debian host. I successfully made my way through the certbot auto process and was happy to see both domains served up securely.

I was so delighted to have achieved this, that I decided to make a full backup of my VM to save this wonderful state of configuration. For this I needed to power off the machine. So I did, copied the VM files to a backup drive and then powered on the VM again.

To my surprise, Apache didn't start. Whereas the post above mentions that for that person Apache did start, yet with errors, mine did not start and got a fatal error instead. The same error: certificate file is missing or empty.

I tried the solution in the post to set higher permissions on the archive dir, as well as the live dir, all to no avail. I tried to inspect the contents of the pem files, but strangely even a cat command told me these files do not exist, even from root privileges.

After doing a ls -lisa on the dir, I noticed the big mistake. These files are not files, they are links. And they were linking to files that do not exist. For example, cert.pem would link to archive/mydomain.com/cert2.pem.

Note the "2" in there! The actual file simply was cert.pem, so after editing the link for all the pem files, it worked again.

It is very well possible that at one point in time, I caused multiple cert files for the same domain. Maybe for the domain and the www sub domain? Maybe because my first attempt failed due to a port issue? I'm not sure what is the reason, but likely it was me.

What does bother me is that a perfectly running setup just breaks from a simple restart. It seems I'm not the only one:

What process can possibly alter a working setup upon a restart? Is it the auto renewal? Something Debian specific?

1 Like
2

According to the design of Certbot, the first certificate saved in an archive directory is cert1.pem (not cert.pem!), the second is cert2.pem, the third is cert3.pem, and so on. The symlink cert.pem in the corresponding live directory should point to the most recent file that exists in the archive directory.

If you have a file called cert.pem instead of cert1.pem in your archive directory, you must have used something other than Certbot, or someone must have manually altered the filenames!

Similarly, the symlinks should never be updated until their targets exist (e.g., cert2.pem is created before the cert.pem link is updated to point to it).

3

Thanks for the response. Sorry, my statement was inaccurate. The actual file in /archive was not cert.pem, indeed it was (and is) cert1.pem. In the live dir, however, the link cert.pem was pointing to /archive/cert2.pem.

cert2.pem does not exist. I did not create that link in any manual way. I have exclusively used the certbot auto process. And as said, the setup was working until I restarted it.

A server restart breaking Apache and invalidating the setup is a big issue. I don’t know what causes it. I restart my server at least once a month and never had such issues, so my gut feeling is that it is related to this specifically. Either way it is worthwhile to think about, because many people hardly ever do a full restart of a server, and may also run into this issue, especially since Debian-based distros are so incredibly common.

Have you got any theories on why a server restart would do such a thing? The only theory I have is that perhaps the auto-renew process triggers something?

4

I don’t know why that would have happened! Would you like to share your logs from /var/log/letsencrypt? Maybe we could find something there that could account for this.

5

Sure I’ll share it. Looks like I produced quite a bit of log entries, 243KB spread over 12 files. Is there a way for me to share it with you privately? I don’t want to publish my domain details openly.

6

You could e-mail to my forum username at eff.org.

7

Done, thanks for looking in this.

8

you need to look into Certificate Transparency

9

Sorry, I don’t know what that means?

10

you made a point about not wanting to share your domains

every time your certificate gets issued by the production server it goes in to a log called certificate transparency

for example here are all the certificates I have issued over the last few months https://crt.sh/?q=%firecube.xyz

The reason why i mention this is that a lot of people who don’t want to share domains should evaluate why and if there is a valid reason be aware of certificate transparency and it’s possible effects.

Andrei

11

Thanks for explaining. My domains are not super secret, it’s just a precaution. I’m using the domains in question not for production, instead for development on a Debian VM which is only powered on when I’m developing from my desktop. That Debian install is a bit old and thus not does not have production-level security. So if I don’t have to share these domains, I won’t.

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.