Lego: badNonce :: JWS has an invalid anti-replay nonce

Romuald · October 10, 2025, 2:14pm

My domain is: poly.tique.info

I ran this command:

ACME_DNS_API_BASE=https://auth.tique.info
ACME_DNS_STORAGE_PATH=/var/lib/lego/poly_tique_info/.lego-acme-dns-accounts.json
EMAIL="MY_EMAIL@example.com"
HOOK="/var/lib/lego/poly_tique_info/scripts/hook.sh"
TXT_RECORD="poly.tique.info"

LEGO_ARGS=(
   --email ${EMAIL}
   --dns "acme-dns"
   --dns.resolvers 9.9.9.9:53
  --domains "poly.tique.info"
)
ACME_DNS_STORAGE_PATH=${ACME_DNS_STORAGE_PATH} \
ACME_DNS_API_BASE=${ACME_DNS_API_BASE} \
/usr/local/bin/lego "${LEGO_ARGS[@]}" renew --renew-hook ${HOOK}

It produced this output:
2025/10/10 15:18:41 acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: Unable to validate JWS :: JWS has an invalid anti-replay nonce: "S5Fh2depWvMecljRf3MIF1WHBbmkwMe1Yhrtmjj2ckcF4dk_cZY"

My web server is (include version): I don't use a webserver. It's dns_01 protocol.

The operating system my web server runs on is (include version): Debian 13.1

My hosting provider, if applicable, is: OneProvider

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
lego version 94b66d7aa92285939a35523253dcebd0980910a1 linux/amd64

I'm trying to renew 3 certificates on different hosts with the same tools and I have been getting this invalid anti-replay nonce for days whatever I try.

I'm using the Joohoi's ACME-DNS as DNS Provider.

I've taken some time to review previous discussions here on that kind of problem but it didn't help.

If that was a global problem as it happened in the past with Let's Encrypt ressources, I would not be alone raising that problem.

Any idea on what I should check on my side is welcomed.

Romuald · October 10, 2025, 2:25pm

Previous query was:

2025/10/10 15:18:38 [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: Unable to validate JWS :: JWS has an invalid anti-replay nonce: "S5Fh2dep2KSkq6of6gKjEl_42eTNhexW_DjRgPZcS4QWyvIDDk4"

So queries are not too slow and nonce should be valid.

MikeMcQ · October 10, 2025, 4:18pm

Can you try adding --log-level debug

lego should be updating the nonce if/as needed. That is a routine operation yet something is clearly going wrong.

And, you are right. If this was a Let's Encrypt server issue we should be seeing many more reports from others about it and we are not. Given it happens on 3 different setups of yours and no other reports it seems likely something common among your installs.

Can you explain more about this version? Which number was it?

Has anything changed about your install of that? Have you tried upgrading to latest lego?

aarongable · October 10, 2025, 4:36pm

For what it's worth, that commit hash does not exist in the upstream lego repository:
https://github.com/go-acme/lego/commit/94b66d7aa92285939a35523253dcebd0980910a1

So I too am very curious about that version of lego.

Romuald · October 11, 2025, 7:19am

Yes. I rebuilt lego with debug code to try to figure out what was wrong.
I'm retrying with an official version and the debug option.

Romuald · October 11, 2025, 7:40am

Lego doesn't use a standard logger and this option does not exist.

I added some custom printf code to try to debug my nonce problem. That's why the code has diverged from the repo. The base code is from Lego version 4.26.0.
With that code I was able to check that nonce where sent as provided by Let's Encrypt and so looked valid.

Here is a start of an exchange with Let's Encrypt service:

Request: GET /directory HTTP/1.1
Host: acme-v02.api.letsencrypt.org
User-Agent: lego-cli/42b6b35167967a6aa4fbd56b669cc3b1a4d5cd23 xenolf-acme/4.26.0 (detach; linux; amd64)
Accept-Encoding: gzip


Response: HTTP/1.1 200 OK
Content-Length: 1065
Cache-Control: public, max-age=0, no-cache
Connection: keep-alive
Content-Type: application/json
Date: Fri, 10 Oct 2025 13:18:20 GMT
Server: nginx
Strict-Transport-Security: max-age=604800
X-Frame-Options: DENY

{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "profiles": {
      "classic": "https://letsencrypt.org/docs/profiles#classic",
      "shortlived": "https://letsencrypt.org/docs/profiles#shortlived (not yet generally available)",
      "tlsclient": "https://letsencrypt.org/docs/profiles#tlsclient",
      "tlsserver": "https://letsencrypt.org/docs/profiles#tlsserver"
    },
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/acme/renewal-info",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "s4p1TWgjpCw": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}

2025/10/10 15:18:20 [INFO] [poly.tique.info] acme: renewalInfo endpoint indicates that renewal is needed
2025/10/10 15:18:20 [INFO] [poly.tique.info] acme: Trying renewal with -1 hours remaining
2025/10/10 15:18:20 [INFO] [poly.tique.info] acme: Obtaining bundled SAN certificate
Request: HEAD /acme/new-nonce HTTP/1.1
Host: acme-v02.api.letsencrypt.org
User-Agent: lego-cli/42b6b35167967a6aa4fbd56b669cc3b1a4d5cd23 xenolf-acme/4.26.0 (detach; linux; amd64)


Response: HTTP/1.1 200 OK
Connection: close
Cache-Control: public, max-age=0, no-cache
Connection: keep-alive
Date: Fri, 10 Oct 2025 13:18:20 GMT
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: P2omldGHufKHtO7Ma74SRZuo8G3qko_TgydT04GGbuIouJkiW3Q
Server: nginx
Strict-Transport-Security: max-age=604800
X-Frame-Options: DENY


Request URI:https://acme-v02.api.letsencrypt.org/acme/new-order
Content {"identifiers":[{"type":"dns","value":"poly.tique.info"}],"replaces":"XXXXXX.YYYYYY"}
Request: POST /acme/new-order HTTP/1.1
Host: acme-v02.api.letsencrypt.org
User-Agent: lego-cli/42b6b35167967a6aa4fbd56b669cc3b1a4d5cd23 xenolf-acme/4.26.0 (detach; linux; amd64)
Content-Length: 567
Content-Type: application/jose+json
Accept-Encoding: gzip

{"payload":"REDACTED PAYLOAD","protected":"eyJhbGciOiJFUzI1NiIsImtpZCI6Imh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMjUyNDM0NDk1MSIsIm5vbmNlIjoiUDJvbWxkR0h1ZktIdE83TWE3NFNSWnVvOEczcWtvX1RneWRUMDRHR2J1SW91SmtpVzNRIiwidXJsIjoiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIn0","signature":"DFDvGAQ7YKw-WwLEPH0B2wVurwQcE2xOYsgTbbhaeWuduLjAp5uraOK705N6GcPAFBFiOzMjKoo2TWoJ05JK3w"}
=============================================================
Base64 decoded protected field:
{"alg":"ES256","kid":"https://acme-v02.api.letsencrypt.org/acme/acct/2524344951","nonce":"P2omldGHufKHtO7Ma74SRZuo8G3qko_TgydT04GGbuIouJkiW3Q","url":"https://acme-v02.api.letsencrypt.org/acme/new-order"}
=============================================================

Response: HTTP/1.1 400 Bad Request
Content-Length: 203
Boulder-Requester: 2524344951
Cache-Control: public, max-age=0, no-cache
Connection: keep-alive
Content-Type: application/problem+json
Date: Fri, 10 Oct 2025 13:18:21 GMT
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: S5Fh2dep5KQAr8vWZ3O1oS-mqoQLzFpMHgvHspAlYYqCJo0eKus
Server: nginx

{
  "type": "urn:ietf:params:acme:error:badNonce",
  "detail": "Unable to validate JWS :: JWS has an invalid anti-replay nonce: \"P2omldGHufKHtO7Ma74SRZuo8G3qko_TgydT04GGbuIouJkiW3Q\"",
  "status": 400
}
2025/10/10 15:18:21 [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: Unable to validate JWS :: JWS has an invalid anti-replay nonce: "P2omldGHufKHtO7Ma74SRZuo8G3qko_TgydT04GGbuIouJkiW3Q"
...

petercooperjr · October 11, 2025, 6:55pm

Hmm. It looks like there's a LEGO_DEBUG_ACME_HTTP_CLIENT environment variable that might be helpful. But there is something very weird going on there. Generally, if a client gets a bad nonce, it should just retry with a new nonce automatically and the user wouldn't even notice. I haven't checked that lego does things that way, but I would have expected so.

MikeMcQ · October 11, 2025, 9:03pm

Sorry about that. I should know better than to rely on a quick Google search

It is odd that the new-order fails right after a new-nonce call

But, lego should be (per its code) retrying the failed new-order.

@Romuald What happens in your log after the new-order fails with the bad nonce? Is it retried and does that one succeed?

Note many staff, and perhaps other volunteers, are away for the long US national holiday. You may not see many comments until Tue.

bruncsak · October 12, 2025, 6:11am

@Romuald, welcome to the community!

Historically, each data center of Letsencrypt had its own nonce cache. It happened, that subsequent connections failed with badnonce error, because one connection was over IPv4 the other one over IPv6, and they were accessing different data centers.

There is no per data center nonce cache any more, there is only one global nonce cache. I do not know the details of the architecture, is it only one dedicated data center that handles all nonce related calls from other data centers, or is it implemented as a distributed database?

mcpherrinm · October 12, 2025, 7:55pm

Every “WFE” (web frontend) can tell where a nonce was issued based on a prefix on the nonce itself, and routes the redemption request to the correct instance to validate it.

There are three reasons a nonce can be invalid:

It is too old, so the nonce server that issued it is no longer tracking it. There’s a “minimum” nonce below which all lower values are rejected. This would typically happen once nonce are over an hour or more old, depending on other nonce redemption traffic.
It has already been redeemed, so the nonce server has recorded a redemption in its in-memory map.
The nonce prefix is from a nonce server which no longer exists, which happens during situations like deploys where we rotate instances. This may affect nonces that are over one minute old, but should be a rare occurrence (deploys happen about weekly).

Romuald · October 14, 2025, 8:00am

@Romuald What happens in your log after the new-order fails with the bad nonce? Is it retried and does that one succeed?
[/quote]

lego do try again with a new nonce received in the bad request response. And it fails again.
lego tries 8 times.

Thanks for that piece of information.

Romuald · October 14, 2025, 8:15am

mcpherrinm:

Every “WFE” (web frontend) can tell where a nonce was issued based on a prefix on the nonce itself, and routes the redemption request to the correct instance to validate it.

There are three reasons a nonce can be invalid:

It is too old, so the nonce server that issued it is no longer tracking it. There’s a “minimum” nonce below which all lower values are rejected. This would typically happen once nonce are over an hour or more old, depending on other nonce redemption traffic.

It has already been redeemed, so the nonce server has recorded a redemption in its in-memory map.

The nonce prefix is from a nonce server which no longer exists, which happens during situations like deploys where we rotate instances. This may affect nonces that are over one minute old, but should be a rare occurrence (deploys happen about weekly).

None of this cases seem to be valid in my case:

Lego requests used the nonce 1 second after it has been received.
Lego does not reuse the nonce as I've been able to log the usage.
I can see a prefix pattern among the nonce received by lego and at least 2 differents prefixes are used. Moreover the process has been failing for days so it's not a valid cause.

I hope there's a bug somewhere on my side causing this behaviour.

Romuald · October 14, 2025, 8:19am

This could be an explanation if other people had the same behaviour that I observed.

MikeMcQ · October 16, 2025, 6:46pm

Is this still a problem for you? Is it still every request that fails?

The only suggestion I have is to try a different ACME Client. Given how early in the ACME API sequence it fails it doesn't matter which Challenge you use. If the challenge fails you know the new-order request succeeded.

Just be sure to use something with a nice enough logging option. Maybe use the APT install of Certbot. It's logging is very good. Normally the Snap install is recommended but for this kind of test the APT install is fine if you don't want to add snap. I believe Trixie has Certbot 4.0 which did have some quirks but shouldn't be a problem for this kind of test. Once installed something like sudo certbot certonly --manual --preferred-challenges=dns -d (domain) should be enough. Its logs are at /var/log/letsencrypt (see: User Guide — Certbot 5.2.0.dev0 documentation)

It is such an unusual problem I am struggling to even think of how that could happen. The results from a different ACME Client might help isolate the problem.

Also, maybe now that we are past the US holiday maybe someone else will have ideas.

giffengrabber · October 16, 2025, 6:59pm

Just spitballing here, could it be a bug in Lego?

MikeMcQ · October 16, 2025, 7:49pm

Well, sure, many things are possible But lego is very popular and I'd think we'd see more reports. It's been nearly a week since this person first noticed it.

I don't have a usable test system to try it out at the moment. The version used here is 4.26 which is most current and came out a month ago. I don't know how lego could have passed any of its own tests with a bug where every nonce failed.

aarongable · October 16, 2025, 8:58pm

On our end we can see that your client has indeed seen a wildly disproportionate number of BadNonce errors. It's getting them at such a high rate - i.e. retrying tens of thousands of time with no backoff -- that it feels like something bigger must be broken. And there are some (several hundred) successes scattered in there. Have you seen any of those on your end?

bruncsak · October 17, 2025, 6:43am

Just a wild guess, like in a brainstorming, sorry if it makes no sense.

What about if multiple instances of lego client running with shared nonce cache?

bruncsak · October 17, 2025, 7:01am

Is there internal debug logging in the ACME server when badNonce error occurs that differentiates between the three cases? Alternatively, is it possible to report the three different cases to the ACME client as detail for badNonce error?

For example, error detailes:

no nonce found, it may have been expired, or never existed
nonce has already been redeemed
no nonce server instance found indicated by the nonce prefix

Romuald · October 17, 2025, 8:49am

Yes. I understood it was not appropriate that is why I stopped it a few days ago.
The lego process is run by a systemd timer which call a systemd service.
When the lego service fail systemd just restart the service. When lego retries it does wait for a random time but there is no backoff. I expect systemd to be able to do some backoff, I will look for that.

For the poly.tique.info domain unfortunately as I have cancelled the service, I do not have any log with journalctl. But for mesure.tique.re I do see there was a partial success for some requests. It failed on my side because of configuration update I had done that were not relevant. This should be fixed by now.

I have made a manual retry for poly.tique.info but I still get badnonce error. So may be the account linked is blacklisted somehow.

I'm going to try with another ACME Client as suggested by MikeMcQ.

Topic		Replies	Views
Regular badNonce errors Client dev	12	2563	May 16, 2018
Getting The client sent an unacceptable anti-replay nonce Help	24	23296	May 2, 2017
Intermittent badNonce errors Help	14	3643	September 16, 2016
JWS has invalid anti-replay nonce Help	11	20056	April 19, 2017
Error renewing certificate from LE: NS returned REFUSED for _acme-challenge Help	40	3837	April 24, 2022

Lego: badNonce :: JWS has an invalid anti-replay nonce

Related topics