Issue with OCSP GET request

My domain is: cbuk-core-testfox2-sf-uksouth.test-cbinnovation.uk

I'm trying to figure out why my newly issued certificate won't verify on Windows with certutil

certutil.exe -verify C:\Temp\cert2.pem
ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)
CertUtil: The revocation function was unable to check revocation because the revocation server was offline.

As far as I can see there is an issue with the OCSP GET request returning unauthorized.
It works fine with a different certificate from last month.

If I construct the request myself using openssl I get the same issue:

$cert1 = 'https://crt.sh/?d=4609675038'
$cert2 = 'https://crt.sh/?d=4769300104'
$r3 = 'https://letsencrypt.org/certs/lets-encrypt-r3.pem'
Invoke-RestMethod $cert1 -OutFile 'C:\Temp\cert1.pem'
Invoke-RestMethod $cert2 -OutFile 'C:\Temp\cert2.pem'
Invoke-RestMethod $r3 -OutFile 'C:\Temp\r3.pem'

& openssl ocsp -noverify -no_nonce -respout C:\Temp\cert1.resp -reqout C:\Temp\cert1.req -issuer C:\Temp\r3.pem -cert C:\Temp\cert1.pem -url "http://r3.o.lencr.org" -text -header "HOST=r3.o.lencr.org" | Out-Null
& openssl ocsp -noverify -no_nonce -respout C:\Temp\cert2.resp -reqout C:\Temp\cert2.req -issuer C:\Temp\r3.pem -cert C:\Temp\cert2.pem -url "http://r3.o.lencr.org" -text -header "HOST=r3.o.lencr.org" | Out-Null

$cert1_path = [System.Net.WebUtility]::UrlEncode(
  [System.Convert]::ToBase64String(
    [System.IO.File]::ReadAllBytes('C:\Temp\cert1.req')
  )
)
$cert2_path = [System.Net.WebUtility]::UrlEncode(
  [System.Convert]::ToBase64String(
    [System.IO.File]::ReadAllBytes('C:\Temp\cert2.req')
  )
)

$response1 = Invoke-WebRequest "http://r3.o.lencr.org/$cert1_path" -Verbose
$response2 = Invoke-WebRequest "http://r3.o.lencr.org/$cert2_path" -Verbose

[System.IO.File]::WriteAllBytes('C:\Temp\cert1.get_resp', $response1.Content)
[System.IO.File]::WriteAllBytes('C:\Temp\cert2.get_resp', $response2.Content)

Write-Host "cert1 response:"
& openssl ocsp -noverify -respin C:\Temp\cert1.get_resp -text
Write-Host "cert2 response:"
& openssl ocsp -noverify -respin C:\Temp\cert2.get_resp -text
VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgMZuS5f8DXyF8pi%2ByoXtowUXg%3D%3D with 0-byte payload
VERBOSE: received 503-byte response of content type application/ocsp-response
VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgOkWi2%2BOVxENXkszitC%2Fj7OQg%3D%3D with 0-byte payload
VERBOSE: received 5-byte response of content type application/ocsp-response

cert1 response:
Produced At: Jun 25 05:29:00 2021 GMT
  Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
      Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
      Serial Number: 0319B92E5FF035F217CA62FB2A17B68C145E
    Cert Status: good
    This Update: Jun 25 05:00:00 2021 GMT
    Next Update: Jul  2 05:00:00 2021 GMT

    Signature Algorithm: sha256WithRSAEncryption
         23:fa:cf:a2:c9:64:c4:0e:82:0c:2c:d2:22:7a:79:02:1f:7f:
         d1:3f:cf:4e:bc:9b:ed:c6:27:26:c0:c9:55:f4:60:93:81:6d:
         da:d5:9b:1e:49:69:ce:d5:6f:4b:ef:8a:12:e1:bf:53:ed:45:
         7b:6d:7f:a0:fb:dc:21:a4:9b:58:72:d3:a5:05:d9:dc:c3:01:
         cc:da:a9:63:44:7f:dd:a2:b8:ad:af:f1:d0:b7:19:6a:69:e3:
         64:b5:f0:65:1c:a5:ed:d8:a2:e5:27:dc:49:38:64:aa:7b:0b:
         b1:19:8f:60:b4:92:ae:b1:65:fd:18:c6:ab:05:60:4b:39:78:
         34:f5:95:94:b3:6c:2a:03:b7:8a:58:53:00:e7:67:dc:68:9a:
         5e:ae:c9:59:f0:0b:87:4f:1a:52:a8:0e:34:84:b3:fe:ab:af:
         fc:a4:a9:47:fe:66:7f:26:8b:2f:73:50:5d:85:29:1a:5e:91:
         9b:be:dc:1a:b1:10:73:2a:e2:6e:9b:05:0b:a4:f0:d2:13:a4:
         65:6b:d6:86:dc:ba:32:3e:e3:1f:e6:c2:52:9c:1b:6b:4b:65:
         a7:d1:86:0c:12:2f:cc:7e:e4:d4:dd:60:86:f2:f2:11:cc:2d:
         cf:e9:e4:fd:d2:fd:85:51:ed:e9:2b:24:90:50:28:15:3e:0e:
         f1:a3:43:87

cert2 response:
Responder Error: unauthorized (6)
2 Likes

Strangely if I don't UrlEncode the path in the get request then the validation is successful:

$cert2_path = [System.Convert]::ToBase64String(
  [System.IO.File]::ReadAllBytes('C:\Temp\cert2.req')
)
$response2 = Invoke-WebRequest "http://r3.o.lencr.org/$cert2_path" -Verbose
[System.IO.File]::WriteAllBytes('C:\Temp\cert2.resp', $response2.Content)
Write-Host "cert2 response:"
& openssl ocsp -noverify -respin C:\Temp\cert2.resp -text | Out-String | Write-Host

VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg+yvTLU/w3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h+vnYsUwsYCEgOkWi2+OVxENXkszitC/j7OQg== with 0-byte payload
VERBOSE: received 503-byte response of content type application/ocsp-response

cert2 response:
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = R3
    Produced At: Jun 27 09:23:00 2021 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
      Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
      Serial Number: 03A45A2DBE395C4435792CCE2B42FE3ECE42
    Cert Status: good
    This Update: Jun 27 09:00:00 2021 GMT
    Next Update: Jul  4 09:00:00 2021 GMT

    Signature Algorithm: sha256WithRSAEncryption
         0f:66:e2:aa:44:eb:6d:9a:c4:32:22:16:78:7f:b3:5d:aa:83:
         39:43:06:6d:0b:a4:da:44:e4:47:4a:59:c3:df:9d:24:de:05:
         56:c2:19:27:9d:11:85:c7:fb:6f:9d:56:3e:2f:8c:94:cb:8a:
         21:dd:55:78:7b:2e:52:f3:b6:94:e7:50:88:1f:f1:ec:5f:a8:
         71:16:cf:86:c1:e8:f2:82:f1:da:3f:3e:89:24:b3:25:57:bd:
         dd:e1:97:46:bd:d7:15:7a:63:47:ff:b2:66:83:2e:63:9a:18:
         c2:60:18:cf:58:b8:4a:fb:dd:a9:14:33:b9:3a:2a:1d:82:1a:
         09:5d:a7:a6:b7:29:c7:b4:99:70:22:4f:f2:69:a4:e5:6e:22:
         18:3a:33:48:1d:58:1b:db:2d:67:c1:16:70:ee:4f:e4:1c:bb:
         1f:d9:52:9f:3f:33:c6:0f:5f:22:db:96:36:ad:94:50:11:f7:
         8e:e0:fa:fc:3a:a0:09:09:15:9c:42:df:6a:42:ed:ea:df:f7:
         d5:e1:d9:cf:84:6b:81:63:95:5f:c7:e9:45:e9:1b:83:2b:64:
         7c:fc:d5:b9:06:d3:60:99:54:c9:6a:b8:11:97:e8:4d:a8:e0:
         f7:cb:89:f0:47:14:5c:69:46:03:68:ea:43:d5:10:10:ab:61:
         51:db:d2:ae

My understanding is the spec requires it to be Url Encoded though

I wonder if it was a bad response from the OCSP server (i.e. Let's Encrypt failed to pre-sign the response for cert2 in time), because your original program runs fine for me right now:

PS /root> ./a.ps1
VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgMZuS5f8DXyF8pi%2ByoXtowUXg%3D%3D with 0-byte payload
VERBOSE: received 503-byte response of content type application/ocsp-response
VERBOSE: GET http://r3.o.lencr.org/MFMwUTBPME0wSzAJBgUrDgMCGgUABBRI2smg%2ByvTLU%2Fw3mjS9We3NfmzxAQUFC6zF7dYVsuuUAlA5h%2BvnYsUwsYCEgOkWi2%2BOVxENXkszitC%2Fj7OQg%3D%3D with 0-byte payload
VERBOSE: received 503-byte response of content type application/ocsp-response
cert1 response:
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = R3
    Produced At: Jun 25 05:29:00 2021 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
      Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
      Serial Number: 0319B92E5FF035F217CA62FB2A17B68C145E
    Cert Status: good
    This Update: Jun 25 05:00:00 2021 GMT
    Next Update: Jul  2 05:00:00 2021 GMT

    Signature Algorithm: sha256WithRSAEncryption
        23:fa:cf:a2:c9:64:c4:0e:82:0c:2c:d2:22:7a:79:02:1f:7f:
        d1:3f:cf:4e:bc:9b:ed:c6:27:26:c0:c9:55:f4:60:93:81:6d:
        da:d5:9b:1e:49:69:ce:d5:6f:4b:ef:8a:12:e1:bf:53:ed:45:
        7b:6d:7f:a0:fb:dc:21:a4:9b:58:72:d3:a5:05:d9:dc:c3:01:
        cc:da:a9:63:44:7f:dd:a2:b8:ad:af:f1:d0:b7:19:6a:69:e3:
        64:b5:f0:65:1c:a5:ed:d8:a2:e5:27:dc:49:38:64:aa:7b:0b:
cert2 response:
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = R3
    Produced At: Jun 27 09:23:00 2021 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 48DAC9A0FB2BD32D4FF0DE68D2F567B735F9B3C4
      Issuer Key Hash: 142EB317B75856CBAE500940E61FAF9D8B14C2C6
      Serial Number: 03A45A2DBE395C4435792CCE2B42FE3ECE42
    Cert Status: good
    This Update: Jun 27 09:00:00 2021 GMT
    Next Update: Jul  4 09:00:00 2021 GMT

    Signature Algorithm: sha256WithRSAEncryption
        0f:66:e2:aa:44:eb:6d:9a:c4:32:22:16:78:7f:b3:5d:aa:83:
        39:43:06:6d:0b:a4:da:44:e4:47:4a:59:c3:df:9d:24:de:05:
        56:c2:19:27:9d:11:85:c7:fb:6f:9d:56:3e:2f:8c:94:cb:8a:
        21:dd:55:78:7b:2e:52:f3:b6:94:e7:50:88:1f:f1:ec:5f:a8:
        71:16:cf:86:c1:e8:f2:82:f1:da:3f:3e:89:24:b3:25:57:bd:
        dd:e1:97:46:bd:d7:15:7a:63:47:ff:b2:66:83:2e:63:9a:18:
        c2:60:18:cf:58:b8:4a:fb:dd:a9:14:33:b9:3a:2a:1d:82:1a:
3 Likes

Yeah, it seems to have started working overnight, possibly the bad response got cached?
Although reading some of the older issues about caching I thought that the responder should get updated before the cert is issued now?

Anyone have any idea on this? Was it a server side/caching issue? Should I leave a delay between the certificate being issued and verifying it?

Hi, @hbuckle,

Thanks for posting about this. Here's an explanation I posted in an older thread about how this could happen: Ocsp server responses for new certificates - #2 by JamesLE

In this case, we did have replication lag during the few minutes when your certificate was issued. I've opened a retroactive incident status report showing the timeframe: Let's Encrypt Status

Sorry about the trouble! We monitor and alert on replication lag, and will keep working on ways to prevent it from affecting OCSP.

It's reasonable for you to expect that a new certificate will immediately have a valid OCSP response available; and, it's also best practice to make sure that your ACME client and integrations will gracefully handle cases when that fails.

7 Likes

@JamesLE - thanks for checking and the explanation. In the future if we suspect replication lag has occured is there any way to remove the cached response from the CDN and get it to query the origin again, rather than having to wait 12 hours for it to expire?

For the future, if you want things to be more fail-save, I would recommend this approach for your deployment process - assuming you can script it/have control over the process:

  1. After the new (renewed) certificate is issued, try to obtain an OCSP response. If this is successful, install the new certificate.
  2. If it is not successful, do not install the new certificate yet - continue using the old one [remember, you still got ~30 days on the clock]. Instead, implement a retry algorithm that periodically (say, every n minutes) retries getting OCSP*.
  3. When OCSP is finally succesful, install the new certificate.

This approach should work nicely with clients that require status checking (or with certs that have must-staple). If your server supports it, OCSP stapling also improves things, but the initial OCSP response must still be obtained before installing the cert to avoid problems.

*I would not assume that it takes the full 12 hours for the OCSP response to be available after errors, therefore some type of periodical retry algorithm is what I would recommend here.

4 Likes

That can't be controlled from your end, unfortunately. We've been thinking about a way to quickly remediate stale cache entries after there's been replication lag - as well as preventing replication lag in the first place - but we aren't there yet. The good news is that Akamai already tends to cache our Unauthorized responses for much less than the headers' full 12 hour value. @Nummer378's suggestion sounds great to me.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.