Staging API responds with 500 internal server error

[dennisjac]

Does the staging API have a problem at the moment? It just responds with an 500 internal server error when I issue the command certbot-auto renew --dry-run -v. The /directory and /acme/new-nonce calls deliver a code of 200 but the /acme/new-order call shows:

https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 500 114
Received response:
HTTP 500
Server: nginx
Content-Type: application/problem+json
Content-Length: 114
Boulder-Requester: 7826991
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: oklwd3QhAh-rM0LV0HNLXfzTp512goS9_QHkHtI9Yf8
Expires: Tue, 09 Jul 2019 14:00:56 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 09 Jul 2019 14:00:56 GMT
Connection: close

{
  "type": "urn:ietf:params:acme:error:serverInternal",
  "detail": "Error creating new order",
  "status": 500
}

[cpu]

Hi @dennisjac, welcome to the community forum.

@dennisjac

Does the staging API have a problem at the moment?

Apologies, this appears to be a side-effect of a change that was activated in the staging environment and subsequently reverted. It looks like orders created during the affected period for a set of names will return 500 errors when creating a new order with the same set of names.

I recommend you delete your staging account ACME credentials and register a new account. That should resolve the errors for you.

Apologies for the service disruption. I'll ask someone on our SRE team to add a note to the staging environment status page.

[cpu]

@cpu

I recommend you delete your staging account ACME credentials and register a new account. That should resolve the errors for you.

I'm pursuing an alternative fix concurrently. I'll provide updates here when possible.

[cpu]

@cpu

I’ll provide updates here when possible.

The status page for the staging env has been updated to mark this event. Thanks again for reporting it @dennisjac. Our monitoring had detected this problem as well but it was misdiagnosed and a communication breakdown prevented a more timely fix.

@cpu

I recommend you delete your staging account ACME credentials and register a new account. That should resolve the errors for you.

Based on the alternative fix this is no longer required. Any orders created during the problematic period in staging will now 404 when accessed directly instead of 500ing. Any orders created by the same account for the same names will produce a new order instead of a 500 or the older broken order.

Please let me know if you run into any other difficulties. We'll be monitoring the situation but I believe it has now been resolved.