OCSP Request failed with following message

Hi,
I am facing the issue only in Europe, as you said. US requests seem to be fine.

As this issue is still ongoing, is there a possible workaround, such as adding the IP of a US load balancer in /etc/hosts ?

I have used this workaround previously with full success. However I would guess that it is not totally reliable because the underlying IPs may be rotated by Akamai at any time.

I've included the vantage point from a Google Cloud US POP below:

ocsp.int-x3.letsencrypt.org. 123 IN     CNAME   ocsp.int-x3.letsencrypt.org.edgesuite.net.              
ocsp.int-x3.letsencrypt.org.edgesuite.net. 1513 IN CNAME a771.dscq.akamai.net.                          
a771.dscq.akamai.net.   19      IN      A       63.243.228.17                                           
a771.dscq.akamai.net.   19      IN      A       63.243.228.10                                           
ocsp.int-x3.letsencrypt.org. 147 IN     CNAME   ocsp.int-x3.letsencrypt.org.edgesuite.net.              
ocsp.int-x3.letsencrypt.org.edgesuite.net. 3563 IN CNAME a771.dscq.akamai.net.                          
a771.dscq.akamai.net.   19      IN      AAAA    2001:5a0:4402::3ff3:e458                                
a771.dscq.akamai.net.   19      IN      AAAA    2001:5a0:4402::3ff3:e45a                                

2 Likes

Just to add another seeing this issue. Based in the UK, and seeing occasional messages as follows:

[Fri Jan 19 08:52:42.177055 2018] [ssl:error] [pid 12429:tid 139904545642240] [client 66.249.64.157:45470] AH01980: bad response from OCSP server: 503 Service Unavailable

We’re still in communications with Akamai support. We’ve made a bit of headway and they’re looking into their European regions still. We’ll update again when there’s something major to report.

2 Likes

The issue is that this is knocking people’s website’s “offline” if they have setup Apache2 with it’s default OCSP stapling settings at least on Firefox (Chrome doesn’t do OCSP iirc, so I don’t know how it’ll react). Refreshing multiple times helps of course, since the server will continue to attempt to get a valid response, but the average user will probably resign before that happens.

There is a workaround available here at least if you don’t use the “OCSP Must Staple” extension: OCSP error is taking down my site in firefox

Sadly the error message from Firefox sugggests a server configuration error, which is strictly speaking not quite correct.

There is no information on https://letsencrypt.status.io/ either, which further obfuscates the source of the issue.

1 Like

I’m curious.

How is it that a simple restart of Nginx resolves the issue? Because for about a month now, the

2018/01/20 10:42:14 [error] 1620#1620: OCSP responder sent invalid “Content-Type” header: “text/html” while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org

error comes up every weekend (Saturday or Sunday), and I have to restart my nginx for everything to start working, until the next week.

Why does a restart fix it?

Hi,
Received the same error in US as well on ‘Sun Jan 21 01:23 PST 2018’.

I have the exact same problem on multiple sites.

As more and more virtual sites of an apache server drop out of the stapling cache, these sites are no longer accessible. The only solution I currently see is to temporarily set SSLUseStapling to off.

Please correct me if I am wrong!

Since 13.01.2018 12:59 our Webserver logs (NGINX Server in Germany) are also beeing floaded with
OCSP responder sent invalid “Content-Type” header: “text/html” while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 2.16.186.27:80

In case anyone is wondering about a temporary and hacky solution, I’ve added to /etc/hosts:

63.243.228.17   ocsp.int-x3.letsencrypt.org

So far, I’ve no errors since 3 days. We need to follow this up (and remove this dirty fix when things go back to normal), so indeed, as K.A.B mentionned, it would be really nice to have some visibility on https://letsencrypt.status.io/

1 Like

It’s safe to say that akamai’s servers are still not OK :frowning:

# cd /etc/letsencrypt/live/status.dogsbody.com
# openssl ocsp -issuer chain.pem -cert cert.pem -text -url http://ocsp.int-x3.letsencrypt.org
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
          Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
          Serial Number: 0356E5FA189305B3BA1BC2C8D13D993D83F8
    Request Extensions:
        OCSP Nonce: 
            041047D59827ABED1246FAB062C7E3A7C30F
Error querying OCSP responder
140474384783000:error:27076072:OCSP routines:PARSE_HTTP_LINE1:server response error:ocsp_ht.c:314:Code=400,Reason=Bad Request

A host lookup on ocsp.int-x3.letsencrypt.org from the affected server…

# host ocsp.int-x3.letsencrypt.org
ocsp.int-x3.letsencrypt.org is an alias for ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net is an alias for a771.dscq.akamai.net.
a771.dscq.akamai.net has address 92.123.64.234
a771.dscq.akamai.net has address 92.123.64.201
a771.dscq.akamai.net has IPv6 address 2a02:26f0:e8::6856:6fb0
a771.dscq.akamai.net has IPv6 address 2a02:26f0:e8::6856:6f88

Please at least update your status page to show that this is an ongoing issue :-/

2 Likes

We have set SSLUseStapling off in our apache config for now and the bad response from OCSP server: 503 Service Unavailable error in the apache logs is gone since then.

I have visited three websites in the last few days where Firefox wouldn’t allow me to access the site due to OCSP being unavailable. I have no control over they setup their servers :-/

I’ve also been seeing a lot of these errors, an example from an Apache errors log from this morning:

[Tue Jan 23 10:08:05.314862 2018] [ssl:error] [pid 23183] AH01941: stapling_renew_response: responder error

I’m using the Mozilla recommended settings:

SSLUseStapling          on
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
SSLStaplingCache        shmcb:${APACHE_RUN_DIR}/ocsp(128000)

Is there a way to improve this configuration in order to mitigate the current situation where the Let’s Encrypt OCSP servers are rather unreliable?

1 Like

Yes, you can turn stapling off in the interim to remove the server’s dependency on the OCSP servers.

SSLUseStapling off

There may be some way to keep stapling on and tune it to deal with the errors better, but I am not sure that an acceptable configuration is possible with the way Apache currently works.

Thanks @_az, if there is no better configuration then I guess disabling it on all our servers is the best option since the only other option would be to advise clients to disable it in their web browsers :frowning:.

Thanks for the suggestion - our operations team has opened a status page incident for this: https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/5a6753733800d404cc4ea7db

I’m hopeful someone will be able to update this thread with more information about the remediation discussions later today.

cc @isk @devnullisahappyplace

Just a thought:
If we disable OCSP stapling on our servers, we would only move the problem to our users.
The browsers/UserAgents would still try to fetch the OCSP response presumingly resulting in the same error.
(please correct me if browsers can handle this in a better fashion)

In one of my servers (doesn’t support stapling) I fetch the OCSP response manually in a file and provide that to my nginx. That process runs as a cron script every hour.

I had failures of SOME requests (not all) at least since 15.1.18. But also some months ago (22.09.17 08:23UTC).
So going with the hypothesis that this is just a load problem, I simply changed the execution-time of my cron script to not-so-defaulty-times, resulting in fewer failures (so far - or maybe you just changed something on your side).

Also: Please PN me if someone knows how browsers implement redundancy in ocsp requests, if an ocsp responder is dead or returns rubbish. Do they try multiple or all hosts behind the cname of the ocsp-uri in our certificates? What would happen if the ca specifies multiple ocsp-uris?

@seanmavley Unfortunately nginx will cache DNS resolutions indefinitely after querying once. Restarting nginx would make it resolve a new IP address for the OCSP server, which may help things.

@cpu @isk @devnullisahappyplace I’m not sure if you’re aware of this nginx behavior, and people are experiencing this with other clients so I doubt stale IPs are responsible, but it could be exacerbating the situation…