ACME client : finalized order stuck on ready state

Hi !

I am currently coding an ACME client using the http-01 and dns-01 challenges. So far I manage to get the certificates for any domains except wildcards one ...

During a wildcard certificate request , after the challenges have been approved I send the finalization request. On normal (not wildcard) domains I get the certificate url but on wildcard ones I m stuck with an order on the ready state.

This is an example of such a reply from the ACME server (here pebble) after a post at the finalization request :

Cache-Control: public, max-age=0, no-cache
Content-Type: application/json; charset=utf-8
Link: https://pebble:14000/dir;rel="index"
Replay-Nonce: pwleNrSD6M25B306lyXykg
Date: Sun, 07 Nov 2021 12:34:46 GMT
Content-Length: 521
{'status': 'ready', 'expires': '2021-11-08T12:34:41Z', 'identifiers': [{'type': 'dns', 'value': 'example.com'}, {'type': 'dns', 'value': '*.example.com'}], 'finalize': 'https://pebble:14000/finalize-order/YGPHisJEosWsPV6LHoIoovcKQ5qw_X9ncZWeimyxKE4', 'authorizations': ['https://pebble:14000/authZ/VkoHu7L_biSgStBTEYyH28rGaTTf5SYrqGmpQDhwOI0', 'https://pebble:14000/authZ/JFb30VGLBO6h_binSHT8C0vb5ITn6zVEl-_X4huFs2I']}

I am a bit confused as I did not expect the behaviour of ACME to change with a wildcard domain. Am I missing something ?

Thank you in advance for your help !! hope my question makes sense :sweat_smile:

1 Like

Hi @salimbeni1 and welcome to the LE community forum :slight_smile:

To be clear: Have you been able to issue certs using DNS-01 - just NOT wildcard certs ?

2 Likes

As the author of the CertSage ACME client, I can state that (when interacting with Boulder) there is absolutely nothing special about handling a wildcard identifier except that only the DNS-01 challenge type is offered for the authorization.

@jvanasco

Any thoughts here about Pebble?

2 Likes

Two questions to troubleshoot:

  • Are you checking the AcmeOrder first to see if it hit "ready" before you call the finalize endpoint? The status may not transition immediately, so you really should poll the Acme Order object until it is ready.

  • Are you checking to ensure there are no errors in the "finalize" response or request? Are you also checking logs to ensure that pebble received the finalize request?

3 Likes

thank you all for your reply !

  • yes , I ve been able to get certificate through dns-01 challenges (when there is no wildcard domain)

  • before sending a post request to the finalize endpoint , I wait for all challenges status to be on the state valid. But what do u mean by polling the Acme order until it is ready ?

Here is an example of the logs of Pebble :

-- for the domains example.com and www.example.com :

pebble_1 | Pebble 2021/11/10 23:44:11 GET /dir -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 POST /sign-me-up -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 There are now 1 accounts in memory

pebble_1 | Pebble 2021/11/10 23:44:11 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 POST /order-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 There are now 1 authorizations in the db
pebble_1 | Pebble 2021/11/10 23:44:11 There are now 2 authorizations in the db
pebble_1 | Pebble 2021/11/10 23:44:11 Added order "LnA5_WD0Lq0nL4IBw28SIIHhtt6vlXm_eEtyeBl1pAw" to the db
pebble_1 | Pebble 2021/11/10 23:44:11 There are now 1 orders in the db
pebble_1 | Pebble 2021/11/10 23:44:11 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 POST /authZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:11 POST /authZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:12 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:12 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:12 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"www.example.com"}, Challenge:(*core.Challenge)(0xc0001be780), Account:(*core.Account)(0xc00013cd80)}
pebble_1 | Pebble 2021/11/10 23:44:12 Starting 3 validations.
pebble_1 | Pebble 2021/11/10 23:44:12 authz O1eHkh9xNdWbaCZ7M8q-jyG_dM6nvDjv6e33bgTZssA set VALID by completed challenge fnmhVF0EaqT0poL-f29lu03cPFNhKeUK-w-DsvebBV8
pebble_1 | Pebble 2021/11/10 23:44:14 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:14 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:14 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:14 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:14 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"example.com"}, Challenge:(*core.Challenge)(0xc0001be5a0), Account:(*core.Account)(0xc00013cd80)}
pebble_1 | Pebble 2021/11/10 23:44:14 Starting 3 validations.
pebble_1 | Pebble 2021/11/10 23:44:14 authz Aos8cjvvZ8ublPnBSoa3qI4Mtavk9raQfcNjOQWmiP4 set VALID by completed challenge 1Zh_ggVB6IgAbbpR0O22bUoevdWGUS_BrkTp6UpQykc
pebble_1 | Pebble 2021/11/10 23:44:16 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:16 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:21 HEAD /nonce-plz -> calling handler()

pebble_1 | Pebble 2021/11/10 23:44:21 POST /finalize-order/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:21 Order LnA5_WD0Lq0nL4IBw28SIIHhtt6vlXm_eEtyeBl1pAw is fully authorized. Processing finalization
pebble_1 | Pebble 2021/11/10 23:44:21 Issued certificate serial 0ec23ceefac65a05 for order LnA5_WD0Lq0nL4IBw28SIIHhtt6vlXm_eEtyeBl1pAw
pebble_1 | Pebble 2021/11/10 23:44:21 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:21 POST /my-order/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:21 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:44:21 POST /certZ/ -> calling handler()

-- and for the domains example.com and *.example.com :

pebble_1 | Pebble 2021/11/10 23:46:34 GET /dir -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:34 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:34 POST /sign-me-up -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:34 There are now 1 accounts in memory
pebble_1 | Pebble 2021/11/10 23:46:34 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:35 POST /order-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:35 There are now 1 authorizations in the db
pebble_1 | Pebble 2021/11/10 23:46:35 There are now 2 authorizations in the db
pebble_1 | Pebble 2021/11/10 23:46:35 Added order "vbPKNFRH9uCDBHJ62M-dLJQjcS6U9dv8QYBOz-x5Xb8" to the db
pebble_1 | Pebble 2021/11/10 23:46:35 There are now 1 orders in the db
pebble_1 | Pebble 2021/11/10 23:46:35 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:35 POST /authZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:35 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:35 POST /authZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:36 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:36 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:36 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"example.com"}, Challenge:(*core.Challenge)(0xc00009db80), Account:(*core.Account)(0xc000088f00)}
pebble_1 | Pebble 2021/11/10 23:46:36 Starting 3 validations.
pebble_1 | Pebble 2021/11/10 23:46:36 authz RVohrufB0lrJmGMnE7B-2nk_Z1T6zLQ5yZfLp1Z_nA8 set VALID by completed challenge I0nIFlrcgm2IXZ_vzy4bU_wbXkJM0x30JhX2eQRf_jk
pebble_1 | Pebble 2021/11/10 23:46:38 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:38 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:38 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:38 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:38 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"example.com"}, Challenge:(*core.Challenge)(0xc00009d9a0), Account:(*core.Account)(0xc000088f00)}
pebble_1 | Pebble 2021/11/10 23:46:38 Starting 3 validations.
pebble_1 | Pebble 2021/11/10 23:46:38 authz Rngjh6ZCGpSnLjIjm48pBTYu0326VIfAdDcwzif_wSk set VALID by completed challenge 0SsheWdIgl6NO5pMDfGb9-4ucRcZ8NvuUqrAEDoLh2k
pebble_1 | Pebble 2021/11/10 23:46:40 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:40 POST /chalZ/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:45 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:45 POST /finalize-order/ -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:45 HEAD /nonce-plz -> calling handler()
pebble_1 | Pebble 2021/11/10 23:46:45 POST /my-order/ -> calling handler()

As you can notice with the wildcard domain name , the certificate is not created. I tried to resend multiple time the post request to the finalize endpoint after the first fail as well as waiting 3seconds but the state of the order remain ready :confused:

1 Like
  • Poll (POST-as-GET) each authorization url (not challenge url) with an empty payload until each authorization status has transitioned to the valid state
  • Submit the CSR to the finalize URL once
  • Poll (POST-as-GET) the order url (not finalize url) with an empty payload until the order status has transitioned to the valid state
  • POST-as-GET the certificate url
2 Likes

After you complete the last challenge, request the AcmeOrder's url and inspect the payload for the AcmeOrder object. Make sure it is in the "ready" state, if it is not, request it once every 1-2 seconds until you see the status change from "pending" to "ready".

The "finalize" operation will not work until the AcmeOrder is "ready". Once you successfully complete all the challenges, the AcmeOrder can transition to "ready", but is not guaranteed to.

Your process should be:

  • post-as-get AcmeAuthorization#1
  • trigger challenge for AcmeAuthorization#1
  • poll AcmeAuthorization#1 until status changes from pending
  • post-as-get AcmeAuthorization#2
  • trigger challenge for AcmeAuthorization#2
  • poll AcmeAuthorization#2 until status changes from pending
  • post-as-get AcmeOrder
  • poll AcmeOrder until status changes from pending

Also, make sure your requests to /finalize are processed correctly with no errors in your client. Pebble will do things like reject 5% of nonces as invalid, to ensure your client can proactively deal with production issues.

Notice that in your logs, the wildcard version does not have a line corresponding to the first order's line:

Order LnA5_WD0Lq0nL4IBw28SIIHhtt6vlXm_eEtyeBl1pAw is fully authorized. Processing finalization

My guess is you may have missed a validation, or you are requesting the finalization before the order object is ready. even if an order has all valid authorizations, pebble may not have gotten to updating the order record as "ready" yet; calling finalization on that order will not work.

I would first troubleshoot this by debugging the request&response to the finalization url. if there is an error in there, you should ensure your client surfaces it.

5 Likes

@jvanasco

Interesting... I don't check for order ready. :thinking:

As soon as I've verified all authorizations valid, I generate the certificate's private key and CSR then submit to finalize url then poll for order valid. Maybe generation lags have been enough to not stumble over finalize before order ready?

4 Likes

It's in part that, and also that Boulder typically handles this much faster than Pebble. I haven't looked at it's code in regards to this, but I would not be surprised if Pebble purposefully introduces a delay on things like this, to mimic possible production lags due to network traffic or server load.

Do you run CI tests against Pebble for CertSage? If not, I really recommend it. Switching to it surfaced dozens of issues for me.

Pebble is ANNOYING but wonderful at the same time, because it simulates errors or error-causing conditions AND offers some slightly different implementations of the ACME spec than Boulder. It really makes you understand the spec, and build clients that can work with multiple providers (even if you only support one, like we do).

5 Likes

Really thank you a lot for your detailed replies !!!

Yes ! this was the main problem
Now that I poll AcmeOrder until status changes from pending the the Acme client works smoothly

5 Likes

@jvanasco, thanks for bringing this into my attention too. Now, it is fixed in the ACME client I am maintaining:

5 Likes

:partying_face: :partying_face: :partying_face:
:tada: :tada: :tada:

3 Likes