OCSP stapling verification timeout revisited

Help
#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: yachats.photos

I ran this command:

acme.sh  --issue  --domain 'yachats.photos' --dns 'dns_gd'  --domain '*.yachats.photos' --dns 'dns_gd'  --home '/tmp/acme/Yachats.Photos/' --accountconf '/tmp/acme/Yachats.Photos/accountconf.conf'

It produced this output: acme.sh successfully expanded the certificate and deployed it properly.

My web server is (include version): Apache/2.4.46 (Ubuntu)

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: Self Hosted

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): acme.sh (thanks neilpang)

This morning I decided to expand my certificate to include a wildcard. Everything worked perfectly. Until I visited the site:

ocsp-error
ocsp-error898×370 14.2 KB

And of course I went to SSL labs and verified my suspicion.

ocsp-error-2

So now I have two certificates and the one requiring stapling is throwing an error and the one that is not requiring stapling works just fine.

ocsp-error-1

All of the other sites hosted on this particular server have stapling enabled and I am guessing that the stapling cache is what is keeping them from throwing errors.

EDIT: I know this is an intermittent issue or at least I believe it is, but I thought I should file a report since this is the first time I actually saw my server itself throw the error and not SSL labs.

1 Like
#2

I wasn't able to discern a specific question you raised or problem you reported, (I'm sure you know what you wanted to say but it didn't end up coming across to me) however, I think what's happening here is as follows:

The Apache httpd doesn't do a great job implementing OCSP Stapling. What you would want a server to do goes like this:

When the server sees it has a new certificate, it immediately fetches the OCSP response for that certificate, and it notes when that response expires. Periodically, and certainly before that expiry, it tries to obtain a newer response. If the response it gets isn't GOOD (either it claims the certificate is revoked, or there's an error) it continues to staple the old, but not yet expired answer, and keeps trying to get a GOOD response.

Together with a daily Certbot or similar run, which (in modern Certbot versions) will check OCSP and if it says EXPIRED try to replace the certificate, this should give excellent results and security.

Unfortunately Apache doesn't do that. Instead (at least when I last looked) it will generally fetch an OCSP response only after the site is first served up to people, and once it has a response, even if it's BAD, it will serve it, apparently believing you'd want your users turned away rather than given an older response for some crazy reason.

If that is indeed the problem here, I'd recommend looking to replace/ upgrade Apache, or at least being aware that this software is in this particular regard not very good.

2 Likes
#3

Really, no question was asked in this thread. Actually I should have tagged it on to the end of:

but didn't think about it in time.

@tialaramex I do appreciate your response, and many others on this form share your views about Apache server. I have considered switching servers although I don't have any real issues with Apache unless this particular incident becomes one of them.

Thank you

2 Likes
#4

I'm not sure that I trust SSL Labs indication here given their track record. I do trust your own findings though, Rip.

1 Like
#5

OK thanks. I am watching. This is the first time MY SERVER has hosed on an OCSP request.
@tialaramex mentions that apache has issues with ocsp requests, but this is the first time I have seen this particular issue after 30 years. (oops dating myself... someone has to.)
Now Rudy (@rg305) has always referred to apache idiosyncrasies. But this wasn't one of them.
Somehow we need to get to the bottom of this. even if it reveals our own lack of knowledge. We're here to learn. At least I am.
So if it is a matter of "switching servers" it has to be warranted. Not just a suspected issue.
I might start a poll.... Which server do you use?" It would be interesting to see which servers are least susceptible to certbot failures or user error.
I used to use openbsd. and it is a great os for lots of stuff.... (invisible firewalls, etc)
Very cool. BUT we want to secure the internet with TLS. Let's do it but
let us do it right. BEST PRACTICE. (EC Council)

2 Likes
#6

See:

2 Likes
#7

Wow so I appreciate the link. Are you using caddy?
What is your experience with whatever you use?
You are a smart person and I can learn from you.
No SH#@

1 Like
#8

I use Caddy for some things, Apache for others. If I'm writing the configuration myself, I'm probably using Caddy--it's much simpler (for one example, my automated Nextcloud installation for FreeNAS took over 1000 lines of Apache config files--with Caddy, it's under 50 lines). And, more relevant to this discussion, it handles all the SSL stuff (including a sensible implementation of OCSP stapling) automatically. OTOH, being written in Go means that any "plugins" need to be included at compile time, so it has its own quirks to deal with.

With respect to OCSP specifically, it's my understanding that Caddy implements stapling much more sensibly than Apache or Nginx, but I don't have much personal experience with either in terms of handling troublesome cases.

3 Likes