Unauthorized trying to fetch OCSP response

Help
1

Hi
Yestarday 1 July 2018 we were having issues fetching OCSP responses that were coming from our aws machines. When same command was executed from our local network the OCSP response would arrive without any issues. This happened for several subdomains under *.luminatesec.com. We were unable to understand why that was happening, and it seemed to resolve by itself after few hours. Is it possible to know if it was something that we did that caused this unauthorized response.

Thanks in advance.

My domain is:
oktatest.luminatesec.com

I ran this command:
curl -o issuer.pem http://cert.int-x3.letsencrypt.org/
openssl x509 -inform der -in issuer.pem -out issuer2.pem
openssl ocsp -issuer issuer2.pem -cert ./oktatest.luminatesec.com.pem -url http://ocsp.int-x3.letsencrypt.org -resp_text

It produced this output:
ocsp: error from server: unauthorized

My hosting provider, if applicable, is:
Amazon EC2 us-west-2

2

Try adding the -header Host="ocsp.int-x3.letsencrypt.org" option to the ocsp command. (Or with a space instead of the = symbol with OpenSSL 1.0.2 or older.)

1 Like
3

Thanks for the reply.

This is unlikely the issue because:

  1. The code of our service which is responsible for fetching the certificate sends the host header for sure
  2. After about 3 hours it started working (same code)
  3. The command was sent with openssl v1.1.0h and it worked fine from local network but didnt work from amazon ec2 network
4

It’s happened in the past, and usually had something to do with the Akamai CDN used by Let’s Encrypt.

It’s best to be tolerant of OCSP outages in your application.

I think you can probably tag someone like cpu or isk if you want them to look into it, but you’ll probably need to bring some extra information like source/destination IPs and exact times.

5

We are adding logic to handle ocps outages. However we would like to know why it happened because its a production system and we need to know why the outage happened if its possible.

It has happened at “July 1st 2018, 19:01:46.466” GMT time zone.

Source IP: 54.245.27.250 (almost certain thats the source ip)

@cpu @isk would love to hear your input on this.
Thanks in advance!

6

I don't have the access required to debug Akamai issues. I've flagged this thread for our operations team to reply to. Thanks,

7

Thanks
We are currently experiencing it again for switchenvtest.luminatesite.com

8

@cpu
I think I found the issue.
My code was doing a GET request to OCSP endpoint ocsp.int-x3.letsencrypt.org in the following fashion: http://ocsp.int-x3.letsencrypt.org/<base64encode(OCSP_REQUEST>). I then sniffed with wireshark and saw that openssl performs a post to ocsp.int-x3.letsencrypt.org with binary OCSP_REQUEST in the body.
The weird thing is that the GET request is working fine with the boulder local test environment but stopped working with the production endpoint.
Any idea if both should be working or only the POST?

9

According to rfc2560 both methods should work but the request should be less than 255 bytes. My request was 315 bytes… is that why it didnt work?

10

Both should be working.

I'm not positive off-hand if this was the cause for your specific case but it sounds like a likely first guess. RFC 6960 indicates:

If HTTP caching is not important or if the request is greater than 255 bytes, the request SHOULD be submitted using POST.

That's only a SHOULD and not a MUST. When our operations team has identified your failing requests it will likely be easier to say conclusively if this was/wasn't the cause.

11

Can you please verify the source IP your requests are coming from, and tell me what IP address(es) your system is/are resolving for ocsp.int-x3.letsencrypt.org? We’ll need that info in order to search for your sessions and troubleshoot the exact cause.

Thanks!

12

Could you please install checkocsp from https://github.com/letsencrypt/boulder/tree/master/test/ocsp/checkocsp and run it on your certificate, from both your local network and your AWS machines?

Also, could you share the contents of issuer2.pem and oktatest.luminatesec.com.pem, from both your local machine and your AWS machine? And the full output from the openssl command on both machines?

13
14

It doesnt happen anymore. The issue resolved by itself after few hours. Once it happens again I will run the checkocsp tool and send the output

15

Great! If it happens again, please don’t forget to also send the files I requested and the OpenSSL output.

Thanks,
Jacob

16

This has been happening on and off for the last 2 days. I would like to understand what is causing it and why does it magically resolve after a while

1 Like
17

Unfortunately I don’t have that data from the date of the issue. I just executed it and got the following responses:

For oktatest.luminatesec.com
nslookup ocsp.int-x3.letsencrypt.org
Server: 172.30.0.2
Address: 172.30.0.2#53

Non-authoritative answer:
ocsp.int-x3.letsencrypt.org canonical name = ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net canonical name = a771.dscq.akamai.net.
Name: a771.dscq.akamai.net
Address: 104.80.88.81
Name: a771.dscq.akamai.net
Address: 104.80.88.113

For switchenvtest.luminatesite.com
nslookup ocsp.int-x3.letsencrypt.org
Server: 172.31.0.2
Address: 172.31.0.2#53

Non-authoritative answer:
ocsp.int-x3.letsencrypt.org canonical name = ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net canonical name = a771.dscq.akamai.net.
Name: a771.dscq.akamai.net
Address: 92.123.72.151
Name: a771.dscq.akamai.net
Address: 92.123.72.176

18

I retrieved Akamai’s logs for those client IPs, edge server IPs, and time windows. I see some successful requests, but no failed requests: they were not logged. I’ll be curious to hear what you find from the other troubleshooting tools. If you’re able to take a packet capture (e.g. with tcpdump) when this is happening, that would be great, and would give us the data we’ll need to engage Akamai engineering if needed.

19

Thanks for the reply. I will wait until the problem happens again and get tcpdump. Meanwhile it seems to be working fine with post method

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.