OCSP Request failed with following message

seanmavley · January 29, 2018, 8:35am

It is strange, except I sailed through this past weekend without any issues. It seems issue has been resolved from my end here. Thanks.

rleeden · January 29, 2018, 10:52am

@isk quick update - no 503s seen since Friday 13:04 UTC for me in the UK.

icing · January 29, 2018, 5:31pm

Hey there,

from the Apache httpd side of things, we are aware that our OCSP implementation is not the best. Recommendations on how to best configure the current functionality can be found here.

Additionally to that, I can recommend to make the stapling cache persistent, using something like

SSLStaplingCache        dbm:ocsp-stapling

in your configuration.

But this all will not help your users when the 503 hits just when Apache wants to renew.

mdhardeman · January 29, 2018, 6:50pm

@icing,

Would you concur, at this time, that deploying OCSP must-staple on Apache is not a current best practice?

I don’t want to distract from the principal issue of this discussion: that the 503s are/were being returned occasionally by the CDN and that this needs/needed a fix. Having said that, in a world of imperfect OCSP responses from numerous CAs, I wanted to clarify best practice for ambitious system admins as things are at the moment.

Thanks,

Matt

isk · January 29, 2018, 8:59pm

@ePhil @rleeden @seanmavley thanks for the feedback on this. I mentioned this is not a permanent fix, but was very helpful in moving us closer to finding the problem.

We will be making some additional changes to try and troubleshoot the root problem which may reintroduce some 503s. Your continued feedback will be appreciated.

isk · January 29, 2018, 10:13pm

Changes have been made and are propagating. Please let us know if you see 503s after 2018-01-29 22:15 UTC.

rleeden · January 30, 2018, 8:01am

@isk no 503s experienced for me in the UK since latest changes made.

ePhil · January 30, 2018, 9:21am

@isk got two 503s (time in UTC):

2018/01/30 01:54:25
2018/01/30 08:16:18

icing · January 30, 2018, 9:28am

I totally think that the current implementation needs to be improved. It is currently not a care-free, best-practise feature to have must-staple on your certificates.

You can do it, but then you have to monitor for OCSP responder errors in your server logs (which I do daily, not that difficult fgrep cron job). If you need safety above the added performance of stapling, turn it off. If you can live with a little risk (and the LE OCSP responders usually work nicely), maybe you turn it on.

Cheers, Stefan

isk · January 30, 2018, 6:24pm

@ePhil I’m watching the failures from the data that we do have at Akamai and it looks like we’re back to or better than previous levels. Please do keep an eye on this (I will be watching for the larger trends to change), but if it stays that low, I’m going to be tapping out on chasing that low of a level of failure.

ePhil · January 31, 2018, 9:27am

@isk well… there were only 4 errors yesterday and while any 500er is bad, I can see your point

Thank you to everyone involved, for fixing this.

isk · January 31, 2018, 9:35am

Thanks for understanding the difficulty of troubleshooting all the vagaries of the internet here.

To be clear, we’re not dropping this on the floor and have at least two more remediations planned in the next few days that should help if something like the spikey/bursty failures we were troubleshooting here comes around. We’re also looking into solutions that will give us better monitoring visibility of the CDN. We don’t want to be serving any 500s with OCSP.

icing · January 31, 2018, 1:02pm

No more OCSP 503 seen from Germany yesterday or today! Good job, guys!

m.sanjay94 · January 31, 2018, 1:10pm

No errors in the last 3 days in EU and US.
Thanks for the fix

mastercool · February 8, 2018, 7:15pm

Just got this new error:
2018/02/08 19:45:11 [error] 10453#10453: OCSP responder sent invalid "Content-Type" header: "application/problem+json" while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org

cpu · February 8, 2018, 7:21pm

This instance was likely due to the maintenance window that was just closed.

mastercool · February 8, 2018, 7:57pm

@cpu That could be. I have not read todays status emails yet.

futureweb · February 22, 2018, 1:31pm

Hope it’s not starting again?

Today in Austria:

[Thu Feb 22 14:07:04.816703 2018] [ssl:error] [pid 12135:tid 140582022260480] (70007)The timeout specified has expired: [client 199.30.231.0:49765] AH01977: failed reading line from OCSP server
[Thu Feb 22 14:07:04.816728 2018] [ssl:error] [pid 12135:tid 140582022260480] [client 199.30.231.0:49765] AH01980: bad response from OCSP server: (none)
[Thu Feb 22 14:07:04.816776 2018] [ssl:error] [pid 12135:tid 140582022260480] AH01941: stapling_renew_response: responder error

[Thu Feb 22 15:07:52.546977 2018] [ssl:error] [pid 11641:tid 140581921548032] (70007)The timeout specified has expired: [client 66.249.66.0:57340] AH01977: failed reading line from OCSP server
[Thu Feb 22 15:07:52.547008 2018] [ssl:error] [pid 11641:tid 140581921548032] [client 66.249.66.0:57340] AH01980: bad response from OCSP server: (none)
[Thu Feb 22 15:07:52.547042 2018] [ssl:error] [pid 11641:tid 140581921548032] AH01941: stapling_renew_response: responder error

system · March 24, 2018, 1:32pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.