OCSP with LetsEncrypt Used to Work But Now Doesn't

[philpennock]

Hey folks; searching the forums, I see that this is a problem which has occurred in the past and the admins have been stomping down on the root-cause triggers each time.

For the Exim MTA project’s mailhub, we use a Let’s Encrypt cert, CN=mx.exim.org, SAN DNS:hummus.csx.cam.ac.uk, DNS:mx.exim.org; we refetch an OCSP staple via a cron job every two days and also immediately after renewing the cert (a monthly cron job).

Since the latest renew, we’re getting Responder Error: unauthorized (6) from openssl ocsp; this is 100% reproducible … reading the previous forum responses, I’m guessing it’s a cached error?

Is this anything we’ve done wrong to trigger it? Would a sleep N between getting the new cert and requesting the first OCSP staple for it help reduce the likelihood of problems? Anything we can do to get this resolved, other than my posting here?

Thanks!
-Phil

[ahaw021]

@philpennock

Hey folks; searching the forums, I see that this is a problem which has occurred in the past and the admins have been stomping down on the root-cause triggers each time.

can you provide some examples of posts so people can review as well

what does your cron job actually do? If possible can you post it here

What is your web server?

Have you made any changes recently?

if it is reproducible can you paste the results here?

Andrei

[philpennock]

Examples:

• https://community.letsencrypt.org/t/ocsp-server-returns-unauthorized-status/21965
• https://community.letsencrypt.org/t/ocsp-error-request-failed-with-ocsp-status-6-http-ocsp-int-x3-letsencrypt-org/24446

and anything else from searching for OCSP unauthorized.

Cron-job takes the existing cert, grabs --ocsp_uri from it, splits the chain into ourcert.pem and issuer.pem and invokes:

# the `-header` flag requires OpenSSL 1.0.2+
$openssl ocsp \
  -header Host "${host_header:?}" \
  -url "${ocsp_uri:?got empty OCSP URI}" \
  -issuer "${TMPDIR:?}/issuer.pem" \
  -cert "${TMPDIR:?}/ourcert.pem" \
  -CAfile "${CAFile:?}" \
  -verify_other "${crtfile}" \
  -no_nonce \
  -respout "$output"

This has been working fine with being invoked every other day for the past couple of months, so the core functionality works fine.

This is not used with a webserver (except for handling the challenge) but with the Exim MTA, as stated. A mail-server, speaking SMTP. No changes have been made recently other than the mentioned cert renewal.

Output is some text from ed splitting the certs (because at two months this is new enough that I’m still watching cron output mails instead of making them less verbose), followed by the exact error I cited in my post:

Responder Error: unauthorized (6)

[ahaw021]

This one is a bit out of my league

reading the articles there seemed to be 2 key issues:

1 was service availability (server not responding at all due to businesses)
2 was a long reply for the OCSP response (longer than what the server was waiting for)

I am assuming you have submitted multiple OCSP request?

If so I am not sure how Boulder and LetsEncrypt deals with these

@jsha - can you chime in here

Andrei

[jsha]

@philpennock

Is this anything we've done wrong to trigger it? Would a sleep N between getting the new cert and requesting the first OCSP staple for it help reduce the likelihood of problems?

For now, a sleep 1 after getting the cert and before requesting OCSP for the first time will help avoid caching issues like this. We're planning to fix this soonish by making the OCSP generation a blocking part of certificate generation.

[philpennock]

@jsha

For now, a sleep 1 after getting the cert and before requesting OCSP for the first time will help avoid caching issues like this. We're planning to fix this soonish by making the OCSP generation a blocking part of certificate generation.

Done, thanks.

In the meantime, the every-two-days crontab renewal succeeded at 17:28 UTC today in picking up a staple generated at 20:00 UTC yesterday, so we have working OCSP and the our OCSP status monitoring stopped whining at me when that succeeded. So this particular incident resolved itself.

(Currently cron, "every two days", to be able to ride out temporary failures, but the cert renewal is a special case where we can't just ride it out because the old one is immediately invalid; somewhere on my long todo list is a standalone service to use DHCP-style timer-driven retries).

[jsha]

Note: If you depend on OCSP stapling, one way to make renewals more robust is to renew the certificate, but if for any reason you can’t get OCSP, keep the old certificate in place until you succeed in fetching OCSP for the new one.