OCSP server sending expired responses + stapling breaks Chrome

How do you do that?

letsencrypt renew


Processing /etc/letsencrypt/renewal/****.com.conf

The following certs are not due for renewal yet:
/etc/letsencrypt/live/****.com/fullchain.pem (skipped)
No renewals were attempted.

I use the Dehydrated client, where you can set RENEW_DAYS in the config file.

With certbot, the official client, have a look at your config file. It looks like you can set something like renew_before_expiry = 1 year to always renew your certs.

Is there a RATE LIMIT on how many CERT I may RENEW from one single IP? (I know there is one for NEW REG)
As we would have to regenerate some thousand CERTs for this … :-/

No. The usual rate limits on certificates per domain apply, though.

Thanks, issue solved for me.

letsencrypt renew --force-renewal

1 Like

The problem IMHO is that the current situation should not be fatal unless you have an EV certificate or you have “Must-staple” set. Neither of this is true in my case and for most of users (I assume). And because of that any OCSP monitoring etc shouldn’t be necessary.

So, I suspect the real problem is a bug most likely in Chrome, because it’s the only browser affected and it isn’t problem either if OCSP stapling isn’t used. Or maybe the the bug in Apache. Can someone more familiar with OCSP stapling related standards confirm?

Choosing to interpret an expired OCSP response as a failure seems eminently reasonable.

If a browser treats a stapled and expired OCSP response as OK, then effectively that browser has no OCSP validation, it almost might as well never look at the OCSP responses at all, since it’s so easy for bad guys to record a valid OCSP response, then play it back forever and fool that browser long after the certificate is marked BAD in OCSP.

Of course it makes no sense for Apache to send expired, or BAD, or otherwise broken OCSP responses over to the client since they can never help, but this is a known shortcoming of the current Apache OCSP implementation and worth complaining about if you’re a server operator using Apache.

1 Like

No browser interprets an expired OCSP response as a failure, seems. The whole problem with this situation is that it’s only Chrome and only a stapled OCSP response.

AFAIK whole OCSP situation is a softfail at the moment which makes sense due to unreliability of OCSP (illustrated perfectly at the moment), OCSP can be blocked, dDOSed etc.

It’s not limited to chrome, we have Java client/server apps with the clients making Rest calls over HTTPS.
Recents JDK do check OCSP validity periods making the applications unusable for the time being.
A workaround is to disable OCSP checking in the JDK preferences but that’s not easy to explain to a customer…

1 Like

Monitoring OCSP status is lovely, but only tells you that it is not working, not that the problem is at Let’s Encrypt.

I ended up at this thread because my monitors started complaining.

Hi all,

Thanks for reporting. We’re looking into the problem and have found a performance problem is causing our OCSP updates to fall behind: https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/584eb241ac0b4a4f0e006291. We’re working to fix it now.

We do have monitoring in place for our OCSP service, but it looks like there is a bug in the monitor that checks only for a successful OCSP response, not expiration.

Sorry for the outage.

2 Likes

Assuming your monitors are complaining e.g. “Warning: OCSP expires in 18 hours” then you were in time to make sure Let’s Encrypt knew they had a problem and were working on a fix. Whereas if your monitors only say “OCSP expired” I would suggest that you need better monitors, because “Aargh, it’s on fire” is better than nothing, but not very much better.

To @jsha I must say, and I admit this is with hindsight, but still, that it makes sense to measure the OCSP system for time-to-drop-dead (in how many hours will the oldest of our current OCSP responses expire?) which tells you how long you have to decide what to do instead if anything goes wrong, and pace (over the course of the 7 day validity period of OCSP certificates, how many times could your infrastructure sign all the certificates at its current rate?) which tells you whether you have enough capacity to continue operations, or need to either shrink operations or buy more capacity.

Totally agreed. It was our intent to measure this, and our monitoring unfortunately fell short. We’ll definitely be making some improvements here.

1 Like

Jacob, thanks for the update.

As a quick, temporary fix for people in this thread: This issue only affects updating of old OCSP responses, not generation of new ones. So if you reissue your certificate you’ll get a fresh response you can staple.

We’re still working as fast as we can to get OCSP signing caught up, but it may take a little while.

Well, this sucks for Must Staple certificates, even with the workaround of issuing new certs :disappointed:

Also confirmed this workaround. Thanks.

I’m now seeing SEC_ERROR_OCSP_OLD_RESPONSE error in Firefox 50.0.2 for my website. This error cannot be dismissed without going into about:config, which makes the website essentially unreachable.

I checked again and my web server is not currently sending any stapled OCSP responses. I’m not using Must-Staple.

@avian2: What’s your site?

Update for everyone: We’re very close to catching up an all expired OCSP. There’s about 47 minutes of catchup left to be done. Again, my apologies for the outage.

1 Like

Thanks a lot for the work…