Unable to meet CA SCT embedding requirements

[Egari]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

I'm placing this on behalf of a service provider who's currently experiencing this issue for multiple domains. To keep this concise, I will limit this to a single domain.

My domain is: http://dmdprints.com

I ran this command:

We use a custom PHP acme client, with POST-as-GET + ACMEv2 capabilities, the following was as a result of a POST-as-GET to the order finalization endpoint.

It produced this output:

HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Thu, 11 Aug 2022 10:17:29 GMT
Content-type: application/problem+json
Content-length: 160
Connection: close
Boulder-requester: <snip>
Cache-control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-nonce: <snip>

{
  "type": "urn:ietf:params:acme:error:serverInternal",
  "detail": "Error finalizing order :: Unable to meet CA SCT embedding requirements",
  "status": 500
}

My web server is (include version): HAProxy/Apache/Php stack

The operating system my web server runs on is (include version): Mixed, various linux OSes with various versions.

My hosting provider, if applicable, is: TransIP B.V.

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): N/A

Note: affected certificate orders are limited to DNS validation, and relate to multiple wildcard + "bare" domain SAN certificates.

Authorizations/Challenges are valid, however, orders seemingly get "stuck" in the processing state after the order finalization attempt errors out with the error output pasted above.

EDIT: I first noted the first errors started appearing around 2022-08-11 20:46 UTC, this seems to be incorrect as the example above occurred at Thu, 11 Aug 2022 10:17:29 GMT.

I checked the status page, but no incidents are logged at this time.

EDIT: we're also seeing newer orders complete successfully, however, the affected orders have remained "stuck" in processing at this time.

[Egari]

Seems I misread the error reporting, the error posted above lists:

Thu, 11 Aug 2022 10:17:29 GMT

as opposed to the later time I had noted in the post.

[schoen]

@lestaff, this sounds like a CA-side issue (although it's especially odd that it should be affecting a single web host rather than everybody). Could someone please take a look?

[petercooperjr]

@Egari

Authorizations/Challenges are valid, however, orders seemingly get "stuck" in the processing state after the order finalization attempt errors out

Can you have your custom client try creating a new order for these domains? I wonder if other more-common clients might be creating a new order with each attempt, and thereby more resilient to a problem where one particular order stops working for some intermittent CA-internal reason like this.

[jcjones]

Looks like one of our two primary datacenters had two 5-minute spikes of latency communicating with roughly half of our configured CT logs, both around the problem time here of 10:17 UTC on 2022-08-11. It looks like it was transient.

If we get extended periods of log embedding issues we change our log publication list in response, but in this case it wasn't long enough of a period to trigger alerts.

[mcpherrinm]

If we fail to submit to CT, that order is dead: We've already issued precertificates, so we can't retry finalizing again. Subsequent calls should get an "already finalized" error.

The client will need to retry and create another order. Re-validating won't be needed since they should be cached still.

[Egari]

Can confirm, new order went through, without revalidating.

Subsequent calls to the finalize endpoint for affected orders did not result in an "already finalized" error, as these got stuck in "processing" and were receiving orderNotReady-type errors.

Thanks for the info!