over the past 20 minutes, we have observed a surge in Cert Renew failures on our Servers, all of which are accompanied by the following error message:
Status 400
{
"type": "urn:ietf:params:acme:error:badNonce",
"detail": "JWS has an invalid anti-replay nonce: \"xxxxxxxxx\"",
"status": 400
}
Since there were no changes on our infrastructure / our Scripts --> This leads us to ponder the potential involvement of the Let's Encrypt (LE) server infrastructure.
Certainly possible that they're doing maintenance or otherwise having issues with their nonce infrastructure. In general, though, clients should just be retrying on getting an invalid nonce error. What client are you using? Is it retrying a bunch and getting that error multiple times?
We are using an extensively customized iteration of lescript within our operational framework.
The instances of renewal errors currently encountered do not impede the functionality of the active certificates, as the existing valid certificates remain effective, and the renewal process will be reattempted at a subsequent juncture.
My intention in highlighting this matter is to raise awareness, in the possible event of issues with the Let's Encrypt servers, given that such behavior diverges from the anticipated norm. (At least I haven't noticed such a huge surge of those Errors out of maintenance windows the last years)
There are currently no scheduled maintenance activities on the Let's Encrypt side; refer to https://letsencrypt.status.io/.
Sorry for the trouble; the issue with cross-datacenter nonce redemption is resolved. I should have filed a status.io notice when I saw it, though the redemption failure rate was (in absolute terms) still really low, only affecting when requests ping-pong between multiple datacenters, and in fact only affecting one direction.
hey @jcjones,
thank you for the insight and the resolution for the matter. The failure rates have already shown signs of normalization on our side, and the renewal process is now achieving successful outcomes once more.
Thank you, and best regards from Austria.
Andy
