Unable to get new certs, "error creating new cert" 500 error

[mnordhoff]

Mine eventually worked.

I should have started a new thread. Though I still can.

[jsha]

Thanks for the reports! We are aware of degraded issuance, and have opened an incident on letsencrypt.status.io while we investigate. Sorry for not doing so sooner.

[waern]

Hi, I’d like to report another instance of the same problem.

We get the same error message:

{
  "type": "urn:acme:error:serverInternal",
  "detail": "Error creating new cert",
  "status": 500
}

from the service

POST https://acme-v01.api.letsencrypt.org/acme/new-cert

We have been using Lets Encrypt to provision certs for domains for a while without such problems. The problems started appearing when we tried to issue certs for some new domains:

  training5.evidentiae.ca
  training6.evidentiae.ca
  training7.evidentiae.ca
  training8.evidentiae.ca
  training9.evidentiae.ca

  api-training5.evidentiae.ca
  api-training6.evidentiae.ca
  api-training7.evidentiae.ca
  api-training8.evidentiae.ca
  api-training9.evidentiae.ca

  upload-training5.evidentiae.ca
  upload-training6.evidentiae.ca
  upload-training7.evidentiae.ca
  upload-training8.evidentiae.ca
  upload-training9.evidentiae.ca

We’re using simp_le, and it aborts on receiving the error from the server. I don’t know on exactly which subset of the above domains the error occurs.

When we removed these domains from the list of domains that we provision using simp_le, the problem disappeared.

I can provide more detailed logs if you want.

Thanks,
David

[schoen]

@jsha, do you know if this is related to the previous problems?

[waern]

We have noticed that the problem happens when adding another new set of domains (named something else than “training-”), and actually the problem might simply be triggered when simp_le notices that some domain has changed and starts provisioning certificates for our entire set of domains. So the problematic request could very well trying to request a certificate for one of our pre-existing domains. What I’m really trying to say is we don’t know which domain that triggers the problem. But in any case we are still seeing this internal error from the Lets Encrypt server today.

[Patches]

@jsha there are also reports of continied 500s in another thread, which might be related.

From @mnordhoff in production:

@mnordhoff

My domain is: mnrd.us, mattnordhoff.net, mattnordhoff.com

I ran this command: sudo -H certbot renew --dry-run

It produced this output: “Attempting to renew cert (mnrd.us) from /etc/letsencrypt/renewal/mnrd.us.conf produced an unexpected error: urn:ietf:params:acme:error:serverInternal :: The server experienced an internal error :: Error finalizing order. Skipping.” (Or the same for other certificates.)

@mnordhoff

I just ran some "certbot renew --dry-run"s. 19 certificates succeeded, 1 failed with “Error finalizing order”.

and with staging from @larryboymi:

@larryboymi

Experienced it today, against staging. It was mostly sporadically but couldn’t place it. After awhile I couldn’t make it happen, but definitely experienced it some.

@larryboymi

Sure @Patches, it was “larry-a.com” which I have for testing purposes (and experienced these errors last weekend). Thanks!

EDIT: I missed that the second reproducer hadn't tried again since last weekend so this one may not be useful (unless it's a totally different issue).

[waern]

We’ve discovered that the error we see happens when we pass the limit of 100 names per certificate. Sorry for the noise.

[mnordhoff]

@waern

We’ve discovered that the error we see happens when we pass the limit of 100 names per certificate. Sorry for the noise.

Precisely what error message do you get? If it's "error creating new cert", it's still a bug that Boulder isn't saying something more specific.

[waern]

Yes, the fact that we got a 500 Internal Error is what made us confused in the first place. The response body was:

{
  "type": "urn:acme:error:serverInternal",
  "detail": "Error creating new cert",
  "status": 500
}

David

[jsha]

Closing the loop, here’s the corresponding Boulder issue: https://github.com/letsencrypt/boulder/issues/3632. Thanks for all who helped out in this thread!