I'm trying to figure out why my newly issued certificate won't verify on Windows with certutil
certutil.exe -verify C:\Temp\cert2.pem
ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)
CertUtil: The revocation function was unable to check revocation because the revocation server was offline.
As far as I can see there is an issue with the OCSP GET request returning unauthorized.
It works fine with a different certificate from last month.
If I construct the request myself using openssl I get the same issue:
I wonder if it was a bad response from the OCSP server (i.e. Let's Encrypt failed to pre-sign the response for cert2 in time), because your original program runs fine for me right now:
PS /root> ./a.ps1
VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgMZuS5f8DXyF8pi%2ByoXtowUXg%3D%3D with 0-byte payload
VERBOSE: received 503-byte response of content type application/ocsp-response
VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgOkWi2%2BOVxENXkszitC%2Fj7OQg%3D%3D with 0-byte payload
VERBOSE: received 503-byte response of content type application/ocsp-response
cert1 response:
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: C = US, O = Let's Encrypt, CN = R3
Produced At: Jun 25 05:29:00 2021 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
Serial Number: 0319B92E5FF035F217CA62FB2A17B68C145E
Cert Status: good
This Update: Jun 25 05:00:00 2021 GMT
Next Update: Jul 2 05:00:00 2021 GMT
Signature Algorithm: sha256WithRSAEncryption
23:fa:cf:a2:c9:64:c4:0e:82:0c:2c:d2:22:7a:79:02:1f:7f:
d1:3f:cf:4e:bc:9b:ed:c6:27:26:c0:c9:55:f4:60:93:81:6d:
da:d5:9b:1e:49:69:ce:d5:6f:4b:ef:8a:12:e1:bf:53:ed:45:
7b:6d:7f:a0:fb:dc:21:a4:9b:58:72:d3:a5:05:d9:dc:c3:01:
cc:da:a9:63:44:7f:dd:a2:b8:ad:af:f1:d0:b7:19:6a:69:e3:
64:b5:f0:65:1c:a5:ed:d8:a2:e5:27:dc:49:38:64:aa:7b:0b:
cert2 response:
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
Version: 1 (0x0)
Responder Id: C = US, O = Let's Encrypt, CN = R3
Produced At: Jun 27 09:23:00 2021 GMT
Responses:
Certificate ID:
Hash Algorithm: sha1
Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
Serial Number: 03A45A2DBE395C4435792CCE2B42FE3ECE42
Cert Status: good
This Update: Jun 27 09:00:00 2021 GMT
Next Update: Jul 4 09:00:00 2021 GMT
Signature Algorithm: sha256WithRSAEncryption
0f:66:e2:aa:44:eb:6d:9a:c4:32:22:16:78:7f:b3:5d:aa:83:
39:43:06:6d:0b:a4:da:44:e4:47:4a:59:c3:df:9d:24:de:05:
56:c2:19:27:9d:11:85:c7:fb:6f:9d:56:3e:2f:8c:94:cb:8a:
21:dd:55:78:7b:2e:52:f3:b6:94:e7:50:88:1f:f1:ec:5f:a8:
71:16:cf:86:c1:e8:f2:82:f1:da:3f:3e:89:24:b3:25:57:bd:
dd:e1:97:46:bd:d7:15:7a:63:47:ff:b2:66:83:2e:63:9a:18:
c2:60:18:cf:58:b8:4a:fb:dd:a9:14:33:b9:3a:2a:1d:82:1a:
Yeah, it seems to have started working overnight, possibly the bad response got cached?
Although reading some of the older issues about caching I thought that the responder should get updated before the cert is issued now?
In this case, we did have replication lag during the few minutes when your certificate was issued. I've opened a retroactive incident status report showing the timeframe: Let's Encrypt Status
Sorry about the trouble! We monitor and alert on replication lag, and will keep working on ways to prevent it from affecting OCSP.
It's reasonable for you to expect that a new certificate will immediately have a valid OCSP response available; and, it's also best practice to make sure that your ACME client and integrations will gracefully handle cases when that fails.
@JamesLE - thanks for checking and the explanation. In the future if we suspect replication lag has occured is there any way to remove the cached response from the CDN and get it to query the origin again, rather than having to wait 12 hours for it to expire?
For the future, if you want things to be more fail-save, I would recommend this approach for your deployment process - assuming you can script it/have control over the process:
After the new (renewed) certificate is issued, try to obtain an OCSP response. If this is successful, install the new certificate.
If it is not successful, do not install the new certificate yet - continue using the old one [remember, you still got ~30 days on the clock]. Instead, implement a retry algorithm that periodically (say, every n minutes) retries getting OCSP*.
When OCSP is finally succesful, install the new certificate.
This approach should work nicely with clients that require status checking (or with certs that have must-staple). If your server supports it, OCSP stapling also improves things, but the initial OCSP response must still be obtained before installing the cert to avoid problems.
*I would not assume that it takes the full 12 hours for the OCSP response to be available after errors, therefore some type of periodical retry algorithm is what I would recommend here.
That can't be controlled from your end, unfortunately. We've been thinking about a way to quickly remediate stale cache entries after there's been replication lag - as well as preventing replication lag in the first place - but we aren't there yet. The good news is that Akamai already tends to cache our Unauthorized responses for much less than the headers' full 12 hour value. @Nummer378's suggestion sounds great to me.