Error renewing certificate from LE: NS returned REFUSED for _acme-challenge

Help
1

Hi,
I'm having issues with the renewal of my certificates using traefik (latest version).
The log files are showing the following error and I can't figure out what is causing it:

time="2022-03-19T19:47:06Z" level=error msg="Unable to obtain ACME certificate for domains \"lan.ooo,*.lan.ooo\" : unable to generate a certificate for the domains [lan.ooo *.lan.ooo]: error: one or more domains had a problem:\n[*.lan.ooo] time limit exceeded: last error: NS ns-cloud-b3.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n[lan.ooo] time limit exceeded: last error: NS ns-cloud-b3.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n" providerName=googleresolver.acme

Can anyone please point me in the right direction?
I can provide further logs if needed.
Thank you so much!

2

No one? :face_with_diagonal_mouth:

3

Do you still get that error? I can't reproduce it.

% for ns in $(dig +short ns lan.ooo); do dig +dnssec @$ns txt _acme-challenge.lan.ooo; done

; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b4.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 30090
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 70 msec
;; SERVER: 216.239.38.107#53(216.239.38.107)
;; WHEN: Mon Mar 21 10:20:47 CET 2022
;; MSG SIZE  rcvd: 568


; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b2.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20411
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 53 msec
;; SERVER: 216.239.34.107#53(216.239.34.107)
;; WHEN: Mon Mar 21 10:20:47 CET 2022
;; MSG SIZE  rcvd: 568


; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b3.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 42354
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 113 msec
;; SERVER: 216.239.36.107#53(216.239.36.107)
;; WHEN: Mon Mar 21 10:20:48 CET 2022
;; MSG SIZE  rcvd: 568


; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b1.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25877
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 46 msec
;; SERVER: 216.239.32.107#53(216.239.32.107)
;; WHEN: Mon Mar 21 10:20:48 CET 2022
;; MSG SIZE  rcvd: 568

%
2 Likes
4

The thing that puzzles me the most is that from the exact same host I am able to resolve the acme challenge correctly.

While my log file contains the following:

time="2022-03-21T09:56:57Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: use dns-01 solver"
time="2022-03-21T09:56:57Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Preparing to solve DNS-01"
time="2022-03-21T09:56:58Z" level=debug msg="legolog: change (Create): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:56:59Z" level=debug msg="legolog: [INFO] Wait for apply change [timeout: 30s, interval: 3s]"
time="2022-03-21T09:56:59Z" level=debug msg="legolog: change (Get): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:02Z" level=debug msg="legolog: change (Get): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:02Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Preparing to solve DNS-01"
time="2022-03-21T09:57:03Z" level=debug msg="legolog: change (Create): {\"deletions\":[{\"kind\":\"dns#resourceRecordSet\",\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:04Z" level=debug msg="legolog: [INFO] Wait for apply change [timeout: 30s, interval: 3s]"
time="2022-03-21T09:57:04Z" level=debug msg="legolog: change (Get): {\"deletions\":[{\"kind\":\"dns#resourceRecordSet\",\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:04Z" level=debug msg="legolog: change (Create): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg\",\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: [INFO] Wait for apply change [timeout: 30s, interval: 3s]"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: change (Get): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg\",\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Trying to solve DNS-01"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Checking DNS record propagation using [1.1.1.1:53 8.8.8.8:53]"
time="2022-03-21T09:57:10Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 3m0s, interval: 5s]"
time="2022-03-21T09:57:10Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T09:57:15Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T09:57:20Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Waiting for DNS record propagation."

I can actually get the proper results by querying the DNS manually:

root@Docker:~# for ns in $(dig +short ns lan.ooo); do dig +dnssec @$ns txt _acme-challenge.lan.ooo; done

; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b4.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58351
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 64     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 64     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 64     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.38.107#53(216.239.38.107)
;; WHEN: Mon Mar 21 09:58:06 UTC 2022
;; MSG SIZE  rcvd: 331


; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b3.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44929
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 64     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 64     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 64     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.36.107#53(216.239.36.107)
;; WHEN: Mon Mar 21 09:58:06 UTC 2022
;; MSG SIZE  rcvd: 331


; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b2.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27519
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 63     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 63     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 63     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.34.107#53(216.239.34.107)
;; WHEN: Mon Mar 21 09:58:07 UTC 2022
;; MSG SIZE  rcvd: 331


; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b1.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5916
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 63     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 63     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 63     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.32.107#53(216.239.32.107)
;; WHEN: Mon Mar 21 09:58:07 UTC 2022
;; MSG SIZE  rcvd: 331
6

Yep, your DNSSEC is fine.

https://dnsviz.net/d/_acme-challenge.lan.ooo/dnssec/

7

I don't know if this can help, but right before it fails I also get the following errors:

time="2022-03-21T10:03:06Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T10:03:11Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T10:03:16Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Cleaning DNS-01 challenge"
time="2022-03-21T10:03:17Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Cleaning DNS-01 challenge"
time="2022-03-21T10:03:18Z" level=debug msg="legolog: [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/authz-v3/89922545260 :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: \"0001_j5-NnkuFE3aX_llcuosRpw9Iy8txDOrr-xd8EfP8yw\""
time="2022-03-21T10:03:19Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/89922545260"
time="2022-03-21T10:03:19Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/89922545270"
time="2022-03-21T10:03:19Z" level=error msg="Unable to obtain ACME certificate for domains \"lan.ooo,*.lan.ooo\" : unable to generate a certificate for the domains [lan.ooo *.lan.ooo]: error: one or more domains had a problem:\n[*.lan.ooo] time limit exceeded: last error: NS ns-cloud-b4.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n[lan.ooo] time limit exceeded: last error: NS ns-cloud-b4.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n" providerName=googleresolver.acme

But that doesn't tell much to me, so any help is much appreciated!

8

Your acme client did messed up. Who developed that?

1 Like
9

If a bad nonce was used, the ACME client should just retry with a new nonce.. Sounds like a badly implemented ACME client to me.

2 Likes
10

This is a Traefik instance, so the acme client used by it is GitHub - go-acme/lego: Let's Encrypt/ACME client and library written in Go

11

Try changing the DNS servers used by Traefik.

1 Like
12

lego retries 10 (!) times before giving up when encountering a badNonce error. So that's probably not it.

2 Likes
13

I think changing DNS is not going to help as the TXT records get properly propagated.

14

Is there any way for me to check what happens there?

15

It might help in validating if they are being propagated.
Or... go at this the long way:

  • Are the authoritative name servers being updated?
  • Are the updates being propagated (to all authoritative name servers)?
  • How long does a full propagation take?
4 Likes
16

Post #4 on this page shows that the records get propagated to all the authoritative NS within 1 minute from the request (TXT records have a TTL of 120 seconds anyway). Even external DNS servers like 8.8.8.8 and 1.1.1.1 get the new values pretty fast, and in the meanwhile I can still see the logs saying "Waiting for propagation".

17

Maybe the system is looking in the "wrong place" to confirm that propagation (since it does exist).
Post #11 asked you to switch DNS servers.

1 Like
18

Hi @rg305, I tried adding 208.67.222.222 and 9.9.9.9 as DNS servers for the validation, but the result is the same.

19

I just tried to take a tcpdump on the system to see the actual traffic going out and I discovered the following:

15:14:07.072652 IP traefik.mydomain.com.55380 > one.one.one.one.domain: 3403+ [1au] NS? mydomain.com. (36)
15:14:07.073559 IP one.one.one.one.domain > traefik.mydomain.com.55380: 3403 4/0/1 NS ns-cloud-b1.googledomains.com., NS ns-cloud-b3.googledomains.com., NS ns-cloud-b4.googledomains.com., NS ns-cloud-b2.googledomains.com. (157)
15:14:07.074368 IP traefik.mydomain.com.38183 > one.one.one.one.domain: 39654+ A? ns-cloud-b1.googledomains.com. (47)
15:14:07.074598 IP traefik.mydomain.com.48711 > one.one.one.one.domain: 16501+ AAAA? ns-cloud-b1.googledomains.com. (47)
15:14:07.081763 IP one.one.one.one.domain > traefik.mydomain.com.38183: 39654 1/0/0 A 216.239.32.107 (63)
15:14:07.082288 IP one.one.one.one.domain > traefik.mydomain.com.48711: 16501 1/0/0 AAAA 2001:4860:4802:32::6b (75)
15:14:07.082937 IP traefik.mydomain.com.49194 > ns-cloud-b1.googledomains.com.domain: 27306 [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:07.083868 IP ns-cloud-b1.googledomains.com.domain > traefik.mydomain.com.49194: 27306 Refused 0/0/0 (41)
15:14:12.085021 IP traefik.mydomain.com.56086 > one.one.one.one.domain: 42309+ [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:12.096160 IP one.one.one.one.domain > traefik.mydomain.com.56086: 42309 4/0/1 TXT "99Y_CmSXEySi3cjUveKw-EtDzvS2HN_hRXQKasWImUE", TXT "iHo0X_Z6vGBlYVzUBKMYOYjp7DPoyu7-nMsLyFC-d9o", TXT "oCxdOrEnGoP6SevZ6sbw9_JRPFtJBYARcZRi7-OCOzQ", TXT "EH_b7fr1pbA5W9toUv8OXc-bpZZn1wE92ajtQDiNS6k" (276)
15:14:12.096499 IP traefik.mydomain.com.53535 > one.one.one.one.domain: 54673+ [1au] NS? mydomain.com. (36)
15:14:12.106983 IP one.one.one.one.domain > traefik.mydomain.com.53535: 54673 4/0/1 NS ns-cloud-b1.googledomains.com., NS ns-cloud-b3.googledomains.com., NS ns-cloud-b4.googledomains.com., NS ns-cloud-b2.googledomains.com. (157)
15:14:12.107846 IP traefik.mydomain.com.42158 > one.one.one.one.domain: 52277+ A? ns-cloud-b1.googledomains.com. (47)
15:14:12.108006 IP traefik.mydomain.com.59332 > one.one.one.one.domain: 3646+ AAAA? ns-cloud-b1.googledomains.com. (47)
15:14:12.108708 IP one.one.one.one.domain > traefik.mydomain.com.42158: 52277 1/0/0 A 216.239.32.107 (63)
15:14:12.108868 IP one.one.one.one.domain > traefik.mydomain.com.59332: 3646 1/0/0 AAAA 2001:4860:4802:32::6b (75)
15:14:12.109297 IP traefik.mydomain.com.44789 > ns-cloud-b1.googledomains.com.domain: 52666 [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:12.110104 IP ns-cloud-b1.googledomains.com.domain > traefik.mydomain.com.44789: 52666 Refused 0/0/0 (41)
15:14:17.110914 IP traefik.mydomain.com.50836 > one.one.one.one.domain: 19355+ [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:17.113157 IP one.one.one.one.domain > traefik.mydomain.com.50836: 19355 4/0/1 TXT "99Y_CmSXEySi3cjUveKw-EtDzvS2HN_hRXQKasWImUE", TXT "iHo0X_Z6vGBlYVzUBKMYOYjp7DPoyu7-nMsLyFC-d9o", TXT "oCxdOrEnGoP6SevZ6sbw9_JRPFtJBYARcZRi7-OCOzQ", TXT "EH_b7fr1pbA5W9toUv8OXc-bpZZn1wE92ajtQDiNS6k" (276)

It seems that Google nameservers are responding with a "Refused".
Question is, why?

20

Obviously, Google nameservers don't think they are authoritative for that zone.
You need to ask 1.1.1.1 which nameservers it sees as authoritative for that zone.

2 Likes
21

Hi @rg305, 1.1.1.1 and 8.8.8.8 both return the same NS:

# dig +short NS mydomain.com @1.1.1.1
ns-cloud-b1.googledomains.com.
ns-cloud-b3.googledomains.com.
ns-cloud-b4.googledomains.com.
ns-cloud-b2.googledomains.com.

# dig +short NS mydomain.com @8.8.8.8
ns-cloud-b1.googledomains.com.
ns-cloud-b3.googledomains.com.
ns-cloud-b4.googledomains.com.
ns-cloud-b2.googledomains.com.

But looking at the pcap file it seems that the problem is not with those 2 DNS servers, they do reply correctly to the acme challenge, what goes wrong is the final check on ns-cloud-b1.googledomains.com cause the "Refused" you see there comes from the nameserver and inspecting the response, I can see the response flags set as "Authoritative: Server is not an authority for domain".
So it seems like the NS doesn't consider itself as authoritative... but it is? I'm confused...