Orders stuck on ready

I'm running my own ACME client written in Ruby. I'm using the staging server.
It was running wonderful and all of a sudden it stopped giving certificate links. All orders, no matter what domain I use, get stuck on the "ready" status.

Authorization link(s) have the status "valid" and finalizing it with the CSR change the order status to "processing" for a moment and then back to "ready". I'm using certificates per 2 domains (www and root domain), or single domains.

My CSR's are valid, checked that too (I use the root domain as a common name and the root plus www domain as AltName). I send them in base64-url-safe DER format. I get no errors over the CSR, just a "processing" state and then back to "ready".

After uploading the CSR, I do a POST-as-GET finalize check and I get this:
"type": "urn:ietf:params:acme:error:orderNotReady",
"detail": "Order's status ("valid") is not acceptable for finalization",
"status": 403

But the order is in the status "ready".

So I downloaded a couple of existing ACME clients, but all do the same: no more certificates for me. All give the exact same behavior.

I checked my server IP for blacklists, it seems listed for email spam, could this be the issue?

I'm moving this to the client dev category, since I suspect whatever you're seeing is related to your custom client. I don't think an order should ever go "backwards" to ready after hitting processing. Can you post some sample code, and the requests/responses you're seeing?

I'm confused how you'd also see it with other ACME clients, what errors are they specifically reporting?

If there was some issue with your specific IP or hostname, you'd get an error explaining the problem. Being on email blocking lists isn't related.

6 Likes

Valid means that a certificate has already been issued (and can be downloaded via the certificate field in the order) - you've already finalized that order in the past. Something is wrong with your client logic, as it appears to double finalize things.

Uploading the CSR is finalization, afterwards you just poll the order object until it's valid/invalid.

Can you provide an example order URL that's stuck in "ready"?

5 Likes

Just to clarify this…

The order transitions to READY when all the authorizations have passed

Once you invoke the FINALIZE endpoint, it transitions to PROCESSING

You should poll the status until it changes, which will either be VALID or INVALID.

Reference the “State Transitions for Order Objects” section

4 Likes

Thanks for your fast response! I have been reading the RFC in detail and coded everything based upon status checks before the next action. I put some output in it so the steps are visible which I will paste here. The 'certificate' link is never there, even after finalizing it. Order status 'ready' means it wants me to finalize again, so thats what I do. Is there any way to download the certificate if the link is never provided in the order? Two days ago there were no problems, the 'certificate' url was listed in the orders, but since 2 days this is no longer the case.

In the logs below I have changed the real domain for example.com:

NewOrder(): Posting new order to: https://acme-staging-v02.api.letsencrypt.org/acme/new-order
NewOrder(): host said:


New order for: ["example.com"]
{
"status": "ready",
"expires": "2024-07-09T23:55:15Z",
"identifiers": [
{
"type": "dns",
"value": "example.com"
}
],
"authorizations": [
"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/13004812363"
],
"finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558415373"
}


Order status: ready, going to finalization url now
DoFinalize(): Sending CSR to https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558415373
DoFinalize(): Answer received from: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558415373


{
"status": "processing",
"expires": "2024-07-09T23:55:15Z",
"identifiers": [
{
"type": "dns",
"value": "example.com"
}
],
"authorizations": [
"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/13004812363"
],
"finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558415373"
}


Sleep for 2 seconds...
DoFinalize(): Doing a Post-as-Get to https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558415373
DoFinalize(): Answer received from: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558415373


{
"type": "urn:ietf:params:acme:error:orderNotReady",
"detail": "Order's status ("valid") is not acceptable for finalization",
"status": 403
}


CheckOrderStatus(): answer received from: https://acme-staging-v02.api.letsencrypt.org/acme/new-order


{
"status": "ready",
"expires": "2024-07-10T00:00:32Z",
"identifiers": [
{
"type": "dns",
"value": "example.com"
}
],
"authorizations": [
"https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/13004812363"
],
"finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/153752113/17558488143"
}


ERROR: CheckOrder() returned unexpected status: ready

1 Like

Just as a FYI, we can see the domain from the order URLs. It does probably make it a bit harder for search engines though.

The problem appears to be that your client confused orders. You have multiple ones and you're apparently mixing them up.

If we look at that order now, available at

https://acme-staging-v02.api.letsencrypt.org/acme/order/153752113/17558415373

We can see that it's "valid" and perfectly waiting for you to download the certificate. Nothing wrong here.

This order on the other hand has never been finalized (order url is here: https://acme-staging-v02.api.letsencrypt.org/acme/order/153752113/17558488143)

The order you're trying to finalize has already been finalized, while the order you're actually wanting to finalize has never been finalized.

5 Likes

Thank you so much! I actually never looked at the numbers in the url.
I'm going to sit on this for a moment to understand it. Thanks again!

2 Likes

Nummer378: You are wonderful! Thank you so so so much!
Its working now!!

2 Likes

No, that can never happen. When you call Finalize, the ACME Order will transition from READY to PROCESSING, and then either VALID or INVALID. The ACME Order will never revert to an earlier status or ask for you to finalize again. If the ACME Server needs you to retry a request, it will be handled with an error code and message in a failure response – however the ACME Order will have been unaffected. Once you call the finalize endpoint successfully, the status will only be one of: PROCESSING, VALID, INVALID. If the status is still READY, you did not successfully call the right endpoint.

So... Your client has some bugs. You are either invoking the wrong ACME Order successfully, or you are invoking the right ACME Order but receiving a failure message and not correctly detecting it.

A really common cause of this is when someone hardcodes a URL or Order number as a test-case while debugging a problem or building out a feature, and then forgets to comment that line out. I have been guilty of this myself.

5 Likes

I was not hardcoding urls, but I have to do that now I guess... :wink:
The order url is never given anywhere and the RFC states something about a mandatory order field in the account object, that wasn't there also. So I got confused. I will now hardcode the url from the finalize url and change "finalize" into "order". Hopefully these url bases never change.

When you create a new order, the order url is given in the Location header. The order url never changes for the lifetime of the order.

Yeah, Let's Encrypt has some divergences from the spec and this is one of them.

8 Likes

This is a bug. After you post the CSR and sleep for a few seconds, you should GET the order URL, not POST-as-GET to the finalize URL again. This will fix the "Order's status ("valid") is not acceptable" error you see.

This is also a bug: checking the order status should be a GET to the order's unique URL, not to the new-order URL. You are accidentally creating another new order here, and that's why you think you're seeing it transition from Processing back to Ready: you're actually creating a second order whose authorizations are already valid.

This is never necessary; all URLs are provided for you in responses. When you create a new order, its unique URL is provided in the Location header.

5 Likes

While I agree with 99 % what you're saying (not poll the finalize URL, but the order URL), RFC 8555 says the polling should also be done with POST-as-GET:

 o  "processing": The certificate is being issued.  Send a POST-as-GET
    request after the time given in the Retry-After header field of
    the response, if any.

(I believe Boulder also accepts GET requests for many endpoints which, according to the RFC should be POST-as-GET, but I think it's prudent to advertise exact RFC requirements instead of deviations from the RFC.)

1 Like

Oh? Thanks, I'll check and change my code right away!

1 Like

:slight_smile: Thanks!

1 Like