OCSP responder returning 503 errors

jvgutierrez · May 15, 2020, 3:29pm

Since 13:00 UTC we’ve been seeing issues trying to fetch OCSP responses against ocsp.int-x3.letsencrypt.org. We are getting intermittent 503s responses from Akamai (HTTP Server header says AkamaiGHost) but apparently everything is ok according to https://letsencrypt.status.io/. There is any ongoing issue with the OCSP responders or the CDN in front of them?

Thanks,
Valentín Gutiérrez

mbanker · May 15, 2020, 3:45pm

We’re seeing the same.

cekrdoso · May 15, 2020, 3:50pm

Yes, same here too…

mklutz · May 15, 2020, 6:13pm

Same. Getting repeated 400 and 503 “Error querying OCSP responder” errors on several servers with no changes on our end.

jillian · May 15, 2020, 7:43pm

Thanks for providing this information. Our team is looking into it. Are there any other details being returned?

jvgutierrez · May 15, 2020, 7:53pm

Thanks @jillian, it looks to be fixed since a few minutes ago, from 19:43:27 UTC seems to be up & running again

server2 · May 15, 2020, 8:01pm

We are still getting this error with many peers behind the ocsp.int-x3.letsencrypt.org like:
23.221.227.164
23.221.227.166
23.221.227.171
23.221.227.172
23.221.227.173
23.221.72.10
23.221.72.11
23.221.72.17
23.221.72.18
23.221.72.19
23.221.72.24
23.221.72.25
23.221.72.26
23.221.72.27

mbanker · May 15, 2020, 8:07pm

It seems to have been fixed a few times today, but its back as of 19:15 UTC.

Errors look like this, all refer to oscp.int-x3.letsencrypt.org:

ssl_certificate.lua:260: set_response_cert(): auto-ssl: failed to set ocsp stapling for XXX - continuing anyway - failed to get ocsp response: OCSP responder returns bad HTTP status code (http://ocsp.int-x3.letsencrypt.org): 503, context: ssl_certificate_by_lua*

server2 · May 15, 2020, 8:10pm

Our error looks like “OCSP responder sent invalid “Content-Type” header: “text/html” while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org”

And the
curl -v http://ocsp.int-x3.letsencrypt.org/

from time to time gives us:

Hostname was NOT found in DNS cache
Trying 23.221.72.9…
Connected to ocsp.int-x3.letsencrypt.org (23.221.72.9) port 80 (#0)

GET / HTTP/1.1
User-Agent: curl/7.38.0
Host: ocsp.int-x3.letsencrypt.org
Accept: /

HTTP/1.1 503 Service Unavailable
Server AkamaiGHost is not blacklisted
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 176
Cache-Control: max-age=0
Expires: Fri, 15 May 2020 20:01:53 GMT
Date: Fri, 15 May 2020 20:01:53 GMT
Connection: keep-alive

<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>
An error occurred while processing your request.<p>
Reference&#32;&#35;102&#46;ec610317&#46;1589572913&#46;236d6a4
</BODY></HTML>

Connection #0 to host ocsp.int-x3.letsencrypt.org left intact

jillian · May 15, 2020, 8:22pm

Thanks for that information @mbanker and @server2. We’ve made a change that should immediately alleviate the errors you’re seeing and will be posting a status.io while we continue to monitor the solution.

mbanker · May 15, 2020, 8:26pm

i can confirm the errors have stopped as of 20:19 UTC. thanks @jillian

server2 · May 20, 2020, 6:55am

Not sure that this is related but for a couple of days from time to time there are errors (for the same set of peers) in log like

OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.172:80
(104: Connection reset by peer) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.166:80
(111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.217.129.90:80

The same errors were before the issue mentioned in the current topic and ended with a serious problem.

server2 · May 22, 2020, 6:12pm

@jillian again, probably not directly related but you should be aware that issue is progressing and there are more and more failures each day

2020/05/22 13:42:12 [error] 6354#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 13:42:12 [error] 6354#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 15:58:12 [error] 6346#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.170:80
2020/05/22 15:58:12 [error] 6346#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.170:80
2020/05/22 15:58:15 [error] 6364#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.170:80
2020/05/22 15:58:15 [error] 6364#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.170:80
2020/05/22 15:59:14 [error] 6351#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.166:80
2020/05/22 15:59:14 [error] 6351#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.166:80
2020/05/22 16:29:14 [error] 6350#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 16:29:14 [error] 6350#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 16:29:18 [error] 6347#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 16:29:18 [error] 6347#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 16:29:19 [error] 6350#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 16:29:19 [error] 6350#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.171:80
2020/05/22 16:50:02 [error] 6362#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.173:80
2020/05/22 16:50:02 [error] 6362#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.173:80
2020/05/22 16:50:04 [error] 6369#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.173:80
2020/05/22 16:50:04 [error] 6369#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.173:80
2020/05/22 16:50:08 [error] 6365#0: recv() failed (111: Connection refused) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.173:80
2020/05/22 16:50:08 [error] 6365#0: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org, peer: 23.221.227.173:80

Topic		Replies	Views
OCSP Request failed with following message Help	78	16960	March 24, 2018
Seem that OSCP servers down again Help	7	1837	August 26, 2017
OCSP responder Problems? (letsencrypt.org DNS Problem) Help	14	9166	September 2, 2018
Ocsp.int-x3.letsencrypt.org is not working in China Help	32	11505	December 15, 2020
Random OCSP timeouts Help	16	4768	May 3, 2019

OCSP responder returning 503 errors

Related topics