SOLVED: Apache's "built-in" OCSP stapling problem

[Rip]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: yachats.photos

I ran this command:

acme.sh --issue --domain 'yachats.photos' --dns 'dns_gd' --domain '*.yachats.photos' --dns 'dns_gd' --home '/tmp/acme/Yachats.Photos/' --accountconf '/tmp/acme/Yachats.Photos/accountconf.conf' --force --reloadCmd '/tmp/acme/Yachats.Photos/reloadcmd.sh' --ocsp-must-staple --log-level 3 --log '/tmp/acme/Yachats.Photos/acme_issuecert.log' 

It produced this output:

Apr 8 03:16:59 	ACME 	52465 	[Thu Apr 8 03:16:58 PDT 2021] Downloading cert.
Apr 8 03:16:59 	ACME 	52465 	[Thu Apr 8 03:16:58 PDT 2021] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/03768a3f8b350590003375dd0f31ae06bc73'
Apr 8 03:16:59 	ACME 	52465 	[Thu Apr 8 03:16:59 PDT 2021] Cert success. 

My web server is (include version): Apache 2.4.?

The operating system my web server runs on is (include version): Ubuntu 18.04.5 LTS

My hosting provider, if applicable, is: Self Hosted

I can login to a root shell on my machine (yes or no, or I don't know): Yup

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): acme.sh

Got this error loading site after cert renewal today:

Turned off " security.ssl.enable_ocsp_must_staple" in firefox to access site.
Based on this KB article:
https://support.mozilla.org/en-US/questions/1149911

So I suppose my question is Whats going on with OCSP verification?

root:toto ~ >> curl -Iki ocsp.root-x1.letsencrypt.org
HTTP/1.1 200 OK
Server: nginx
Content-Length: 0
Cache-Control: max-age=37233
Expires: Fri, 09 Apr 2021 13:08:50 GMT
Date: Fri, 09 Apr 2021 02:48:17 GMT
Connection: keep-alive

root:toto ~ >> curl -Iki ocsp.int-x3.letsencrypt.org
HTTP/1.1 200 OK
Server: nginx
Content-Length: 0
Cache-Control: max-age=5869
Expires: Fri, 09 Apr 2021 04:29:03 GMT
Date: Fri, 09 Apr 2021 02:51:14 GMT
Connection: keep-alive

root:toto ~ >> curl -Iki ocsp.int-x4.letsencrypt.org
HTTP/1.1 200 OK
Server: nginx
Content-Length: 0
Cache-Control: max-age=35258
Expires: Fri, 09 Apr 2021 12:40:06 GMT
Date: Fri, 09 Apr 2021 02:52:28 GMT
Connection: keep-alive

Ideas?

[tialaramex]

You've asked for certificates which tell relying parties (visitors to your web site, or their web browser at least) that you will definitely do OCSP Stapling and they mustn't trust the site if they don't see a stapled OCSP response.

Your web server, apparently, does not provide the OCSP response. So, although you've given us lots of diagnostics about other things. What we're going to need to know here is about the web server. Which server software do you use? Maybe Apache? Nginx? Has it worked previously with this exact configuration? If not, what did you change? (If you don't remember, say "I don't remember", don't guess). What do the server's logs say about this OCSP stapling?

[Rip]

Please read my answers to the questionnaire.
Server version: Apache/2.4.46 (Ubuntu) to be more specific.

I have been "stapling" certs on this server for years, which is why I posted the question.
It is fair to say though that the issue does not exist in any other flavour of browser. Only Firefox.

@tialaramex

What do the server's logs say about this OCSP stapling?

Checking the logs now.

[tialaramex]

@Rip

Server version: Apache/2.4.46 (Ubuntu) to be more specific.

Apache's "built-in" OCSP stapling implementation was historically total garbage and unsafe to use. If you are using mod_md then that's safe, because it was written by somebody who understood what they were doing, 2.4.46 should have mod_md but I can't tell from what you've posted that you're using it and I suspect not.

You are of course free to keep using the built-in OCSP stapling but stuff like this will just sometimes happen, and since there's a decent implementation now (in mod_md) it hardly seems worth yelling at the Apache developers for doing a bad job at this point.

Oh, and yes I checked your site was actually broken, this has nothing to do with Firefox other than that Firefox actually implements OCSP Must Staple and so it rejects the broken site unless you explicitly configure it to carry on unsafely regardless.

[Rip]

@tialaramex Thank you. I appreciate your insight.
I wasn't familiar with mod_md until recently and am taking steps to either enable it or move to Nginx.
Apache also has other issues that has me considering a move.
EDIT: As mentioned mod_md does ship with Apache/2.4.46. and I shall implement it based on your comments. (Always learning). Thanks again.

[Osiris]

Now you're sending a certificate without the must-staple feature. And also no stapled OCSP response. Which of course is also sort of a "solution".

[schoen]

@tialaramex

Apache's "built-in" OCSP stapling implementation was historically total garbage and unsafe to use.

One problem that I remember with the built-in implementation (which I think is very possibly related to this) is that it would happily throw away old, valid OCSP status caches under various circumstances. I think it would do this (1) whenever Apache was restarted, and (2) whenever it attempted to download a new OCSP status, regardless of whether it succeeded. Of course, behaviors like those would enormously increase the risk of stapling failures.

[JuergenAuer]

Hi @Rip

@Rip

Got this error loading site after cert renewal today:

that happens always if I start the server-daten test system (port 442, not public visible) after some hours / days inactivity with a certificate with "Must staple".

That's an IIS 10.

The application starts, but the IIS 10 doesn't have something cached.

So sending back the result to the client -> that error.

Parallel: Checking OSCP.

F5 to refresh - all works.

Same with a local installation:

MOZILLA_PKIX_ERROR_REQUIRED_TLS_FEATURE_MISSING

It's not really a problem, it's feature-specific.

Apache may have the same problem.

[Rip]

It's true. I had to put a band-aid on the server whilst I resolve the real issue. Which still exists.
Of course it really isn't an issue for Lets Encrypt, it is definitely an Apache issue... so I apologize. The certificate and/or the process of obtaining it was NEVER a problem for me. What is puzzling for me is that there are a whole bunch of vhosts on the same server that aren't complaining or throwing errors at all. Configs are identical relating to the certs and stapling. Maybe they are caching at different intervals?
Thank you @schoen , @JuergenAuer , @Osiris for confirming what @tialaramex pointed out is most likely something I was apparently reluctant to admit.

EDIT: mod_md is now enabled globally.