Trying to understand urn:acme:error:badNonce

connaryscott · September 24, 2019, 4:23am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

This just started to happen today and we haven’t made any changes

Our domain name for this invocation is: local-debug-4.qc.frg.tech

we have been given rate limit exception for a few domains including frg.tech, frgcloud.com, and frganalytics.com.

we just started getting them today not sure why.

This is the dehydrated client.

I ran this command:

./letsencrypt.sh --ca 'https://acme-v01.api.letsencrypt.org/directory' --challenge  'dns-01' --cron --domain  'local-debug-4.qc.frg.tech'

It produced this output:
ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)

Details:
{
“type”: “urn:acme:error:badNonce”,
“detail”: “JWS has no anti-replay nonce”,
“status”: 400
}

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

Here is some debut output:

DEBUG curl -s -w "%{http_code}" -o "/var/folders/k2/z51zlk0x3kb3x3lcdpfpcjsm8bwm7y/T/XXXXXX.YdplmwRj" "https://acme-v01.api.letsencrypt.org/acme/new-authz" -d "{"header": {"alg": "RS256", "jwk": {"e": "AQAB", "kty": "RSA", "n": "5b88awrA7h-WfEfJSt0oQ8EcL7jIytywM_isOhoRNZBYgagGH3RHAetKKu2koMi8DiKf-NSmnTwj6y7HQWStP5uAnnRHJhw4AeEL32gPp28oGVuurEZVGbj8sdJkjNzqlty_qPGwpgUnH6_1jSFyu8lhBffUX2re4AjZv0WgOOqu2tOyd6gMvLp7-rsjwxFz_fPGrib2VUStRJzgIYKLyioflbXnt9ZpjHHhGeXLW6-NPvwH_ovDpYzNwIqRLo_bXRmTaLQrCTFw9vAB1PxaGTomFA1x2QOFS6eWKfleAYz3_fJ9KzZPhEkrYkMg1spw2_0YE6beB2OdYz-BzsVnF9LZv8Q2uxaBQA0CYjH3_rpl7h0wEp1igZjlYIQfzObAfGyAcr3vVkCYEkz8k9BjqeL77Lh8KSjGoSQZTmIXE-5IYvXsXHRLG6J2I2lH6Zbj9H07SinylsiTmDdfc3Fm1fft094WGQa7GSxj836U5POrRKvxq-fyWGbhn_zWkJnYnQpW4jiYKU_t-FCtYkxRanFvjZ1C0GVcYzvGhuTVVzCZaw1w3lXdTpA6ve2D-JbtIi8jnVl0_o0II12RWhpW8bB9N1dhLUC5dQGTTRfkYD4FwYa_Z6mHXOfTVf_MT7Ml1wuQlp47gQL6V0NtPv5cGF1WazDNq_qHJPTs455LXsE"}}, "protected": "eyJhbGciOiAiUlMyNTYiLCAiandrIjogeyJlIjogIkFRQUIiLCAia3R5IjogIlJTQSIsICJuIjogIjViODhhd3JBN2gtV2ZFZkpTdDBvUThFY0w3akl5dHl3TV9pc09ob1JOWkJZZ2FnR0gzUkhBZXRLS3Uya29NaThEaUtmLU5TbW5Ud2o2eTdIUVdTdFA1dUFublJISmh3NEFlRUwzMmdQcDI4b0dWdXVyRVpWR2JqOHNkSmtqTnpxbHR5X3FQR3dwZ1VuSDZfMWpTRnl1OGxoQmZmVVgycmU0QWpadjBXZ09PcXUydE95ZDZnTXZMcDctcnNqd3hGel9mUEdyaWIyVlVTdFJKemdJWUtMeWlvZmxiWG50OVpwakhIaEdlWExXNi1OUHZ3SF9vdkRwWXpOd0lxUkxvX2JYUm1UYUxRckNURnc5dkFCMVB4YUdUb21GQTF4MlFPRlM2ZVdLZmxlQVl6M19mSjlLelpQaEVrcllrTWcxc3B3Ml8wWUU2YmVCMk9kWXotQnpzVm5GOUxadjhRMnV4YUJRQTBDWWpIM19ycGw3aDB3RXAxaWdaamxZSVFmek9iQWZHeUFjcjN2VmtDWUVrejhrOUJqcWVMNzdMaDhLU2pHb1NRWlRtSVhFLTVJWXZYc1hIUkxHNkoySTJsSDZaYmo5SDA3U2lueWxzaVRtRGRmYzNGbTFmZnQwOTRXR1FhN0dTeGo4MzZVNVBPclJLdnhxLWZ5V0diaG5feldrSm5ZblFwVzRqaVlLVV90LUZDdFlreFJhbkZ2aloxQzBHVmNZenZHaHVUVlZ6Q1phdzF3M2xYZFRwQTZ2ZTJELUpidElpOGpuVmwwX28wSUkxMlJXaHBXOGJCOU4xZGhMVUM1ZFFHVFRSZmtZRDRGd1lhX1o2bUhYT2ZUVmZfTVQ3TWwxd3VRbHA0N2dRTDZWME50UHY1Y0dGMVdhekROcV9xSEpQVHM0NTVMWHNFIn0sICJub25jZSI6ICIifQ", "payload": "eyJyZXNvdXJjZSI6ICJuZXctYXV0aHoiLCAiaWRlbnRpZmllciI6IHsidHlwZSI6ICJkbnMiLCAidmFsdWUiOiAibG9jYWwtZGVidWctNC5xYy5mcmcudGVjaCJ9fQ", "signature": "NT0DWRJ4zUw9aBBK2gbaBg7JaXTLTxo8SQHbi3VSvx2HxKJU8y3hUL4fpHzYqSxKE_PpoefLlDa4VMUPrDZUg7G7p9U96BFcZ45AgSjRbgQpIIttWi-GtNLxONzUJ1s5posan6j9TwA4PbKZm0RtC4ScgC9mRPDPDKMpVlWffLZcUp0JLtxzQTsZi6iRFZRdIZ2tphJJuVR8q5AehynQ89XL9BcprufVwq7hPMgBc34NzYx1JgCT5h0M7Ha1o5TeJsNLs_f1vWdN7O2i-osc9IxHycJL45-Pba_HTcP7ImqpXx_TgjOyllqkwWhTkP-kQSGQX1w07ACZLE517oGdCKo66KviT_H7IF_pSTh6YK4mhY0cRohJS6sOx-sey8pUfqe4gO4ISNFueitvvTm092En3X8KgVFeaqouKkecxrvv3LeE69XVBPIo78EgK-1VuXXnk9ikDZh6E3Fs9aAwb_jrXe9-QIh1iJtGxiP7StmLfirjPtPG7yZNMjMKOgOpjeQBNWydIiSpnSaGtfGLqyYEW-we4zEMFSxgIjB1PA6OjPnpWRDePdcV5IRgAiWgS55L8erNpMHdQRUvSX1xp_5sH_OXZ5jUeXGZ06NjUZQB6niP8U0faJObJF3AIN9xc_eFOiVXO9lj8qwC98WSh7GRHCiyXxLAhENFYEJRdhA"}"
  + ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)

mnordhoff · September 24, 2019, 4:29am

Well, the CA thinks that the client's request is missing one field.

Can you answer those questions?

If you're not using the latest version of Dehydrated, can you upgrade it?

It looks like there was at least one bug like this fixed some time ago:

github.com/dehydrated-io/dehydrated

JWS has no anti-replay nonce

opened 03:19PM - 09 May 18 UTC

closed 07:03PM - 09 May 18 UTC

FlorentCoppint

Hi, I use Dehydrated to deploy certificates in a private environment (local B…oulder install). I have no problem except today on a server which was already configured with 1 domain in domains.txt, I added a second one. And Dehydrated throws me that error for this second domain : > # INFO: Using main config file /etc/dehydrated/config > Processing git2.priv.company.net > + Checking domain name(s) of existing cert... unchanged. > + Checking expire date of existing cert... > + Valid till Jun 30 23:41:35 2018 GMT Certificate will not expire > (Longer than 30 days). Skipping renew! > Processing docker.priv.company.net > + Signing domains... > + Generating private key... > + Generating signing request... > + Requesting authorization for docker.priv.company.net... > + ERROR: An error occurred while sending post-request to https://acme-v01.api.priv.company.net/acme/new-authz (Status 400) > > Details: > HTTP/2 400 > boulder-requester: 1005 > cache-control: public, max-age=0, no-cache > content-type: application/problem+json > replay-nonce: Sl0aksakzPWJZN2Xtocr6o5Ybh6UIz1cF1vN5KyR9F8 > content-length: 100 > date: Wed, 09 May 2018 15:13:53 GMT > > { > "type": "urn:acme:error:badNonce", > "detail": "JWS has no anti-replay nonce", > "status": 400 > } > Do I hit a "bug" when first domain does not need renew and second one is new ? Tell me which information do you need to debug this. Boulder is up-to-date, and functional (signed certificates this morning) and I just upgraded dehydrated to 0.6.2 with same error. Thank you.

Edit:

Let's Encrypt made this change today:

Maybe it somehow changed what HTTP version your client is negotiating with the server?

ronaldgetz · September 24, 2019, 7:23am

It happens on our server too, just started from today. What can we do to solve this?

mnordhoff · September 24, 2019, 7:27am

@connaryscott:

It shouldn’t have anything to do with this issue, but why are you using --ca to use the acme-v01 API? Why not use acme-v02?

@ronaldgetz:

Please answer the questions below:

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

ronaldgetz · September 24, 2019, 7:36am

Hi this is my environment for your reference.

My domain is: order.nicethingstoshare.com

I ran this command: I am using Openresty with lua-resty-auto-ssl (https://github.com/GUI/lua-resty-auto-ssl)

It produced this output:
2019/09/24 07:11:51 [error] 2298#2298: 5 [lua] ssl_certificate.lua:97: issue_cert(): auto-ssl: issuing new certificate failed: dehydrated failure, context: ssl_certificate_by_lua, client: 27.96.112.134, server: 0.0.0.0:443
2019/09/24 07:11:51 [error] 2298#2298: 5 [lua] ssl_certificate.lua:286: auto-ssl: could not get certificate for order.nicethingstoshare.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_lua, client: 27.96.112.134, server: 0.0.0.0:443
2019/09/24 07:11:54 [error] 2298#2298: *8 [lua] lets_encrypt.lua:41: issue_cert(): auto-ssl: dehydrated failed: env HOOK_SECRET=fbd53ef559204eaded0ce66f4958eaba3674d5c272aa982057bb12cd3b43e70d HOOK_SERVER_PORT=8999 /usr/local/openresty/luajit/bin/resty-auto-ssl/dehydrated --cron --accept-terms --no-lock --domain order.nicethingstoshare.com
–challenge http-01 --config /etc/resty-auto-ssl/letsencrypt/config --hook /usr/local/openresty/luajit/bin/resty-auto-ssl/letsencrypt_hooks status: 256 out: # INFO: Using
main config file /etc/resty-auto-ssl/letsencrypt/config
Processing order.nicethingstoshare.com

Signing domains…
Generating private key…
Generating signing request…
Requesting authorization for order.nicethingstoshare.com…
err: + ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)

Details:
{
“type”: “urn:acme:error:badNonce”,
“detail”: “JWS has no anti-replay nonce”,
“status”: 400
}

My web server is (include version): Openresty Nginx

The operating system my web server runs on is (include version): Amazon Linux release 2 (Karoo)

My hosting provider, if applicable, is: AWS EC2

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): I am not sure but I followed this blog last week https://www.storyblok.com/tp/automatic-ssl-multi-tenant

_az · September 24, 2019, 7:37am

grep "VERSION=" /usr/local/openresty/luajit/bin/resty-auto-ssl/dehydrated | head -n 1

ronaldgetz · September 24, 2019, 7:48am

The result is
VERSION="0.5.0"

_az · September 24, 2019, 7:49am

Great. I can 100% reproduce the issue at 0.5.0:

alex at x1 in ~/Downloads/dehydrated (tags/v0.5.0)
$ ./dehydrated -6 -f config -c -x
# INFO: Using main config file config
Processing 80755380.ngrok.io
+ Checking domain name(s) of existing cert... unchanged.
+ Checking expire date of existing cert...
+ Valid till Dec 23 06:36:00 2019 GMT Certificate will not expire
(Longer than 30 days). Ignoring because renew was forced!
+ Signing domains...
+ Generating private key...
+ Generating signing request...
+ Requesting challenge for 80755380.ngrok.io...
  + ERROR: An error occurred while sending post-request to https://acme-staging.api.letsencrypt.org/acme/new-authz (Status 400)

Details:
{
  "type": "urn:acme:error:badNonce",
  "detail": "JWS has no anti-replay nonce",
  "status": 400
}

Upgrade to 0.6.5.

ronaldgetz · September 24, 2019, 7:50am

excuse me, but how to update the dehydrated to 0.6.5? Since it comes with openresty, I am not sure how.

_az · September 24, 2019, 7:51am

That is going to be a question that you will need to pose to the openresty project.

What I would try doing is replacing the contents of /usr/local/openresty/luajit/bin/resty-auto-ssl/dehydrated with the contents of https://raw.githubusercontent.com/lukas2511/dehydrated/v0.6.5/dehydrated .

It should work, assuming there were no breaking changes between 0.5.0 and 0.6.5.

_az · September 24, 2019, 7:55am

There is also a second, less invasive change you can try.

On line 485 (of 0.5.0) you will find this line:

nonce="$(http_request head "${CA}" | grep Replay-Nonce: | awk -F ': ' '{print $2}' | tr -d '\n\r')"

If you add a -i after the grep, like below:

nonce="$(http_request head "${CA}" | grep -i Replay-Nonce: | awk -F ': ' '{print $2}' | tr -d '\n\r')"

it should also fix your problem.

The underlying change that triggered this is probably the move to Cloudflare and HTTP/2 that happened today, where headers are canonicalized to lower case (at least, that’s the way curl outputs them).

But you should really upgrade to 0.6.5.

mnordhoff · September 24, 2019, 7:57am

Akamai really didn’t support/enable HTTP/2?

_az · September 24, 2019, 7:59am

Nup.

$ curl --http2 --head --resolve acme-v02.api.letsencrypt.org:443:104.110.202.18 https://acme-v02.api.letsencrypt.org/acme/new-nonce
HTTP/1.1 200 OK
Server: nginx
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0101siuLcOwmHeOtqKhbcTM9R6_4RkrTGdSCHL8sdZgT5aQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Content-Length: 0
Expires: Tue, 24 Sep 2019 07:59:28 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 24 Sep 2019 07:59:28 GMT
Connection: keep-alive

ronaldgetz · September 24, 2019, 8:01am

This solution fixed the issue!!! Thank you so much @_az!

notes: I tried to update to version 0.6.5 but it breaks openresty so updating this one line works!

mnordhoff · September 24, 2019, 8:03am

That's not good.

Do current versions of OpenResty still package an old version of Dehydrated?

Is OpenResty aware of this issue yet?

ronaldgetz · September 24, 2019, 8:25am

Not sure about that, but their build keep failing on their Github. They should aware of this issue.

connaryscott · September 24, 2019, 11:47am

Yes I can confirm that this one line change grep -i Replay-Nonce also resolved my issue.

I have to admit that I never would have found this in a reasonable amount of time. Thank you for your quick response.

elizilbershtein · September 24, 2019, 10:42pm

In order to make it work I had to patch line 281 as well
grep -i Location: || true

from
REAL_LICENSE="(http_request head "{CA_TERMS}" | (grep Location: || true) | awk -F ': ’ '{print 2}' | tr -d '\n\r')" to REAL_LICENSE="(http_request head “${CA_TERMS}” | (grep -i Location: || true) | awk -F ': ’ ‘{print $2}’ | tr -d ‘\n\r’)"

GUI · September 30, 2019, 6:24am

lua-resty-auto-ssl v0.13.0 is now out, which should solve the issue (it upgrades the bundled version of Dehydrated to 0.6.5). Sorry for the trouble! If you’re still seeing any issues, please let us know over on the lua-resty-auto-ssl issue tracker.

SEI_Dan · October 8, 2019, 5:39pm

Thank you _az - I’ve been banging my head against a wall for a week.

In my case I am running a vm appliance using Debian 9 and the latest dehydrated package is 0.3.1-3+deb9u2 (at least from the default repos.)

I added the first listed “grep -i” fix from your post. The second “-i” couldn’t be entered because the line didn’t exist.

I am now able to update my SSL certificate.
Thank you! - now to update Debian