OCSP error is taking down my site in firefox

I’m getting a SEC_ERROR_OCSP_TRY_SERVER_LATER error in firefox which is preventing the site from loading at all. This has happened a few times in the past, but goes away eventually.

I’m running apache 2.4 on Centos 7. The server is manage by cpanel and i’m gettitng the certificate via their AutoSSL option.

Is there anything I can do to fix this, or is this an issue with the OCSP server?

Thanks!

1 Like

It may be difficult to inspect this because it’s managed from cpanel, but is your Apache doing OCSP stapling?

Because having web browsers talk to an OCSP server for every new connection has negative privacy implications and can hurt reliability (sort of as you’re seeing here) there is a feature called OCSP stapling, where the OCSP response instead of being fetched from a server by each web browser, is fetched from the OCSP server periodically by the web server, and then it “staples” that to the certificates it sends over, so browsers get the same information (signed by the CA) but don’t have to make another connection to get it.

However, with OCSP stapling problems that used to definitely be with the OCSP server, now might be trouble on your web server. Apache is definitely capable of fetching a “try later” response and then stapling that to certificates and sending those over, which is, -ugh- I don’t know how they thought that option was helpful.

If you definitely have, or think you might have, OCSP stapling enabled, then short of disabling it, which may not be possible from cpanel, you could try restarting the Apache server itself if that’s possible and then try the site a couple of time, leaving a few seconds between each try ? I think that when it’s restarted it will fetch a fresh OCSP response and so it might get a good response this time.

Still awkward though, really all the major web servers need to do better at OCSP stapling. Nobody wants an expired or invalid OCSP response, and nobody wants to wait a few seconds after each restart for it to work properly.

1 Like

It looks like it's on in the httpd.conf:

<IfModule ssl_module>
   # cipher and protocol directives can be set in WHM under 'Apache Configuration' -> 'Global Configuration'
    SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AE$
    SSLProtocol All -SSLv2 -SSLv3
    SSLPassPhraseDialog  builtin

    <IfModule socache_shmcb_module>
        SSLUseStapling on
        SSLStaplingCache shmcb:/etc/apache2/run/stapling_cache_shmcb(256000)

        # Prevent browsers from failing if an OCSP server is temporarily broken.
        SSLStaplingReturnResponderErrors off
        SSLStaplingErrorCacheTimeout 60
        SSLSessionCache shmcb:/etc/apache2/run/ssl_gcache_data_shmcb(1024000)
    </IfModule>
    <IfModule !socache_shmcb_module>
        SSLSessionCache dbm:/etc/apache2/run/ssl_gcache_data_dbm
    </IfModule>

    SSLSessionCacheTimeout  300
    Mutex                   file:/etc/apache2/run ssl-cache
    SSLRandomSeed startup builtin
    SSLRandomSeed connect builtin

    # The Listen port can be updated using 'Tweak Settings' -> 'System',
    # However, if you have any Apache Reserved IPs, then this Tweak setting will
    # be ignored. Instead, each IP on your system (excluding Apache Reserved IPs)
    # will be listed here.
    Listen 0.0.0.0:443
    Listen [::]:443

    AddType application/x-x509-ca-cert .crt
    AddType application/x-pkcs7-crl .crl
</IfModule>

It might have negative effects if the web server itself can't reach the OCSP server. With the client observing the provided OCSP stapled response (or the lack thereof), it invalidates the connection. If the client would try to get the response itself, it would probably (unless configured otherwise) forgive the lack of response.

That happened to me earlier this summer when LE put all the OCSP servers into maintenance at the same time. One (or more) of my sites were, of course, effectively offline because there were no valid OCSP responses to verify my certificates.

And that's why non-stapled OCSP is worthless. And that's why there's something called "OCSP Must Staple" (implementable in Let's Encrypt certificates!).

I support OCSP Must Staple. Just pointing out the worthless effect is handy when the OCSP servers are (all) down, although I'd prefer not having to rely on that effect.

According to my settings, it looks like OCSP stapling is on. Is there anyway to confirm? It looks like the OCSP servers are down again since my site won’t currently load.

It's quite up and running from my end :slight_smile:

Tested with openssl ocsp -issuer chain.pem -cert cert.pem -text -verify_other chain.pem -url http://ocsp.int-x3.letsencrypt.org/ -header "Host" "ocsp.int-x3.letsencrypt.org":

OCSP Response Data:
    OCSP Response Status: successful (0x0)

et cetera..

1 Like

Still getting the same trylater error message here.

Also tested with openssl s_client -connect 321pool.net:443 -tls1 -tlsextdebug -status |grep OCSP -A 2 -B 1:

OCSP Response Data:
    OCSP Response Status: trylater (0x3)
1 Like

When that happens, also try the direct ocsp query like I did above. That way, we can find out if it’s a fault in Let’s Encrypt OCSP server or with your stapeling.

You can also try the site

which tries a bunch of revocation systems, including different modes of OCSP

1 Like

According to that site, it looks like the server stapled an invalid response?

https://certificate.revocationcheck.com/321pool.net

According to that site, it looks like the server stapled an invalid response?

@mike817 Yes

If your certificate does not have the "must-staple" attribute, a temporary fix could be to disable OCSP Stapling.

Are you referring to turning off the check in firefox?

No, the OCSP Stapling server feature, so your server doesn’t risk sending invalid OCSP answer.

Aha. Apache documents that it will “fake” a tryLater response if its own OCSP resolver doesn’t succeed by default.

“Default: SSLStaplingFakeTryLater on”

Again I have no idea who thought this is helpful. But it could mean that the problem is with some network configuration for your server, which forbids it from accessing the OCSP server even though that server works for everybody else.

The easiest fix may be to disable stapling. Or at least set SSLStaplingReturnResponderErrors off so that when the Apache server has no good answers it shuts up rather than stapling errors.

In some sense a “better” fix may be to investigate and fix the network configuration. But you may have restricted network access for good reasons.

3 Likes

That setting got the server back online for now. Does that mean the OCSP check is in the browsers hands now? Also as far as network issues go, which process is making the stapling request? I'd like to run it directly from the server and see if anything is blocking it.

You mean, other than the user running Apache?

Yes, browsers will now contact the OCSP server themselves if they are inclined to (whether they bother, and if so whether they just shrug it off if the OCS server doesn’t respond, is often configurable)

The Apache httpd is making an OCSP request exactly like the one a browser would make, a correct response is signed by either the exact same issuer as the original certificate (old school, now frowned on) or by another issuer owned by the same CA just for answering OCSP queries (safer). It should get an OK response, and staple that. As we’ve seen, an OK response is available but somehow your server doesn’t get one. So probably if you’re able to run the openssl ocsp test on the server where Apache lives, you’ll see if there’s a problem.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.