Error renewing certificate from LE: NS returned REFUSED for _acme-challenge

onixid · March 19, 2022, 7:53pm

Hi,
I'm having issues with the renewal of my certificates using traefik (latest version).
The log files are showing the following error and I can't figure out what is causing it:

time="2022-03-19T19:47:06Z" level=error msg="Unable to obtain ACME certificate for domains \"lan.ooo,*.lan.ooo\" : unable to generate a certificate for the domains [lan.ooo *.lan.ooo]: error: one or more domains had a problem:\n[*.lan.ooo] time limit exceeded: last error: NS ns-cloud-b3.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n[lan.ooo] time limit exceeded: last error: NS ns-cloud-b3.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n" providerName=googleresolver.acme

Can anyone please point me in the right direction?
I can provide further logs if needed.
Thank you so much!

onixid · March 21, 2022, 9:03am

No one?

9peppe · March 21, 2022, 9:23am

Do you still get that error? I can't reproduce it.

% for ns in $(dig +short ns lan.ooo); do dig +dnssec @$ns txt _acme-challenge.lan.ooo; done

; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b4.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 30090
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 70 msec
;; SERVER: 216.239.38.107#53(216.239.38.107)
;; WHEN: Mon Mar 21 10:20:47 CET 2022
;; MSG SIZE  rcvd: 568


; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b2.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20411
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 53 msec
;; SERVER: 216.239.34.107#53(216.239.34.107)
;; WHEN: Mon Mar 21 10:20:47 CET 2022
;; MSG SIZE  rcvd: 568


; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b3.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 42354
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 113 msec
;; SERVER: 216.239.36.107#53(216.239.36.107)
;; WHEN: Mon Mar 21 10:20:48 CET 2022
;; MSG SIZE  rcvd: 568


; <<>> DiG 9.16.11 <<>> +dnssec @ns-cloud-b1.googledomains.com. txt _acme-challenge.lan.ooo
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 25877
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; AUTHORITY SECTION:
lan.ooo.                300     IN      SOA     ns-cloud-b1.googledomains.com. cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300
lan.ooo.                300     IN      RRSIG   SOA 8 2 21600 20220410085945 20220319085945 3100 lan.ooo. ay6I7YtWibddasmBkoxtb5+OLjooVOibU2E92M+KQzPmqEqj6OKTfHNK 4AB2ntpPvCht52puut3ZxQeo72vDSaIlBGZGBmehBUt4CPzKubyofMrq MCPh6YCCLXkRH6M2mIbX5xyDRh0r+0InZPh8nyTa/bdYtxTpdWlYtTSB 2J0=
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN NSEC3 1 0 1 66BF67CF890814B2 GH02GA1KBBEDC2DTGJS3LTT9EDT85NPR NS SOA MX RRSIG DNSKEY NSEC3PARAM CDS
negbrf21orbnuq5v0cgh6v976b4bs8df.lan.ooo. 300 IN RRSIG NSEC3 8 3 300 20220410085945 20220319085945 3100 lan.ooo. nOglqDE1J9e6JKPJIrf8RpWEoD/PpuBPikS4itRsjhYasaOrcg7MHuTE +zaE/dzvynUSbAmTcOTQe9dJiv0DnDHMaueI4ynQ/mFvQVCyeqQv5cXl y6b3VPOQFe/RiRsSfiLOhdJWUvSLXsG0Kq/U2yBUJ1j58ymqd3Ti2Tyj xK0=

;; Query time: 46 msec
;; SERVER: 216.239.32.107#53(216.239.32.107)
;; WHEN: Mon Mar 21 10:20:48 CET 2022
;; MSG SIZE  rcvd: 568

%

onixid · March 21, 2022, 10:00am

The thing that puzzles me the most is that from the exact same host I am able to resolve the acme challenge correctly.

While my log file contains the following:

time="2022-03-21T09:56:57Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: use dns-01 solver"
time="2022-03-21T09:56:57Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Preparing to solve DNS-01"
time="2022-03-21T09:56:58Z" level=debug msg="legolog: change (Create): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:56:59Z" level=debug msg="legolog: [INFO] Wait for apply change [timeout: 30s, interval: 3s]"
time="2022-03-21T09:56:59Z" level=debug msg="legolog: change (Get): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:02Z" level=debug msg="legolog: change (Get): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:02Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Preparing to solve DNS-01"
time="2022-03-21T09:57:03Z" level=debug msg="legolog: change (Create): {\"deletions\":[{\"kind\":\"dns#resourceRecordSet\",\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:04Z" level=debug msg="legolog: [INFO] Wait for apply change [timeout: 30s, interval: 3s]"
time="2022-03-21T09:57:04Z" level=debug msg="legolog: change (Get): {\"deletions\":[{\"kind\":\"dns#resourceRecordSet\",\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:04Z" level=debug msg="legolog: change (Create): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg\",\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: [INFO] Wait for apply change [timeout: 30s, interval: 3s]"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: change (Get): {\"additions\":[{\"name\":\"_acme-challenge.lan.ooo.\",\"rrdatas\":[\"k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg\",\"S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw\"],\"ttl\":120,\"type\":\"TXT\"}]}"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Trying to solve DNS-01"
time="2022-03-21T09:57:05Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Checking DNS record propagation using [1.1.1.1:53 8.8.8.8:53]"
time="2022-03-21T09:57:10Z" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 3m0s, interval: 5s]"
time="2022-03-21T09:57:10Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T09:57:15Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T09:57:20Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Waiting for DNS record propagation."

I can actually get the proper results by querying the DNS manually:

root@Docker:~# for ns in $(dig +short ns lan.ooo); do dig +dnssec @$ns txt _acme-challenge.lan.ooo; done

; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b4.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58351
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 64     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 64     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 64     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.38.107#53(216.239.38.107)
;; WHEN: Mon Mar 21 09:58:06 UTC 2022
;; MSG SIZE  rcvd: 331


; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b3.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44929
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 64     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 64     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 64     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.36.107#53(216.239.36.107)
;; WHEN: Mon Mar 21 09:58:06 UTC 2022
;; MSG SIZE  rcvd: 331


; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b2.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27519
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 63     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 63     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 63     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.34.107#53(216.239.34.107)
;; WHEN: Mon Mar 21 09:58:07 UTC 2022
;; MSG SIZE  rcvd: 331


; <<>> DiG 9.16.27-Debian <<>> +dnssec @ns-cloud-b1.googledomains.com. txt _acme-challenge.lan.ooo
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5916
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1472
;; QUESTION SECTION:
;_acme-challenge.lan.ooo.       IN      TXT

;; ANSWER SECTION:
_acme-challenge.lan.ooo. 63     IN      TXT     "k76vwsWhhiQy_VjsAEfo2aOv7EQZ8Qjd-kVZZdzthOg"
_acme-challenge.lan.ooo. 63     IN      TXT     "S47r-a9jKBhEyYbdcmSrN2BCqiGwBzuelMBOX2exGyw"
_acme-challenge.lan.ooo. 63     IN      RRSIG   TXT 8 3 120 20220410085945 20220319085945 3100 lan.ooo. MrSVyIBXbXDZBfXgNGd8qYskZ7JY8I0kRMLrTwcdPtIZjqWG6/b4oVwu b75jrSVtSGgYIS3RWuA6aNHqmzDVU7cuumeU+zMqfjepzL+yvC6JHeXa oITg4ZMcD9XEYvyU9onn2p+VlF9ufbMkSwgFVcPF2oSVKo67UQw6Q0YB N6w=

;; Query time: 0 msec
;; SERVER: 216.239.32.107#53(216.239.32.107)
;; WHEN: Mon Mar 21 09:58:07 UTC 2022
;; MSG SIZE  rcvd: 331

9peppe · March 21, 2022, 3:48pm

Yep, your DNSSEC is fine.

https://dnsviz.net/d/_acme-challenge.lan.ooo/dnssec/

onixid · March 21, 2022, 4:04pm

I don't know if this can help, but right before it fails I also get the following errors:

time="2022-03-21T10:03:06Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T10:03:11Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Waiting for DNS record propagation."
time="2022-03-21T10:03:16Z" level=debug msg="legolog: [INFO] [*.lan.ooo] acme: Cleaning DNS-01 challenge"
time="2022-03-21T10:03:17Z" level=debug msg="legolog: [INFO] [lan.ooo] acme: Cleaning DNS-01 challenge"
time="2022-03-21T10:03:18Z" level=debug msg="legolog: [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/authz-v3/89922545260 :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: \"0001_j5-NnkuFE3aX_llcuosRpw9Iy8txDOrr-xd8EfP8yw\""
time="2022-03-21T10:03:19Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/89922545260"
time="2022-03-21T10:03:19Z" level=debug msg="legolog: [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/89922545270"
time="2022-03-21T10:03:19Z" level=error msg="Unable to obtain ACME certificate for domains \"lan.ooo,*.lan.ooo\" : unable to generate a certificate for the domains [lan.ooo *.lan.ooo]: error: one or more domains had a problem:\n[*.lan.ooo] time limit exceeded: last error: NS ns-cloud-b4.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n[lan.ooo] time limit exceeded: last error: NS ns-cloud-b4.googledomains.com. returned REFUSED for _acme-challenge.lan.ooo.\n" providerName=googleresolver.acme

But that doesn't tell much to me, so any help is much appreciated!

9peppe · March 21, 2022, 4:06pm

Your acme client did messed up. Who developed that?

Osiris · March 21, 2022, 5:11pm

If a bad nonce was used, the ACME client should just retry with a new nonce.. Sounds like a badly implemented ACME client to me.

onixid · March 21, 2022, 5:32pm

This is a Traefik instance, so the acme client used by it is GitHub - go-acme/lego: Let's Encrypt/ACME client and library written in Go

rg305 · March 21, 2022, 7:02pm

Try changing the DNS servers used by Traefik.

Osiris · March 21, 2022, 7:16pm

lego retries 10 (!) times before giving up when encountering a badNonce error. So that's probably not it.

onixid · March 21, 2022, 7:50pm

I think changing DNS is not going to help as the TXT records get properly propagated.

onixid · March 21, 2022, 7:51pm

Is there any way for me to check what happens there?

rg305 · March 21, 2022, 9:32pm

It might help in validating if they are being propagated.
Or... go at this the long way:

Are the authoritative name servers being updated?
Are the updates being propagated (to all authoritative name servers)?
How long does a full propagation take?

onixid · March 22, 2022, 8:09am

Post #4 on this page shows that the records get propagated to all the authoritative NS within 1 minute from the request (TXT records have a TTL of 120 seconds anyway). Even external DNS servers like 8.8.8.8 and 1.1.1.1 get the new values pretty fast, and in the meanwhile I can still see the logs saying "Waiting for propagation".

rg305 · March 22, 2022, 2:47pm

Maybe the system is looking in the "wrong place" to confirm that propagation (since it does exist).
Post #11 asked you to switch DNS servers.

onixid · March 22, 2022, 3:04pm

Hi @rg305, I tried adding 208.67.222.222 and 9.9.9.9 as DNS servers for the validation, but the result is the same.

onixid · March 22, 2022, 3:16pm

I just tried to take a tcpdump on the system to see the actual traffic going out and I discovered the following:

15:14:07.072652 IP traefik.mydomain.com.55380 > one.one.one.one.domain: 3403+ [1au] NS? mydomain.com. (36)
15:14:07.073559 IP one.one.one.one.domain > traefik.mydomain.com.55380: 3403 4/0/1 NS ns-cloud-b1.googledomains.com., NS ns-cloud-b3.googledomains.com., NS ns-cloud-b4.googledomains.com., NS ns-cloud-b2.googledomains.com. (157)
15:14:07.074368 IP traefik.mydomain.com.38183 > one.one.one.one.domain: 39654+ A? ns-cloud-b1.googledomains.com. (47)
15:14:07.074598 IP traefik.mydomain.com.48711 > one.one.one.one.domain: 16501+ AAAA? ns-cloud-b1.googledomains.com. (47)
15:14:07.081763 IP one.one.one.one.domain > traefik.mydomain.com.38183: 39654 1/0/0 A 216.239.32.107 (63)
15:14:07.082288 IP one.one.one.one.domain > traefik.mydomain.com.48711: 16501 1/0/0 AAAA 2001:4860:4802:32::6b (75)
15:14:07.082937 IP traefik.mydomain.com.49194 > ns-cloud-b1.googledomains.com.domain: 27306 [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:07.083868 IP ns-cloud-b1.googledomains.com.domain > traefik.mydomain.com.49194: 27306 Refused 0/0/0 (41)
15:14:12.085021 IP traefik.mydomain.com.56086 > one.one.one.one.domain: 42309+ [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:12.096160 IP one.one.one.one.domain > traefik.mydomain.com.56086: 42309 4/0/1 TXT "99Y_CmSXEySi3cjUveKw-EtDzvS2HN_hRXQKasWImUE", TXT "iHo0X_Z6vGBlYVzUBKMYOYjp7DPoyu7-nMsLyFC-d9o", TXT "oCxdOrEnGoP6SevZ6sbw9_JRPFtJBYARcZRi7-OCOzQ", TXT "EH_b7fr1pbA5W9toUv8OXc-bpZZn1wE92ajtQDiNS6k" (276)
15:14:12.096499 IP traefik.mydomain.com.53535 > one.one.one.one.domain: 54673+ [1au] NS? mydomain.com. (36)
15:14:12.106983 IP one.one.one.one.domain > traefik.mydomain.com.53535: 54673 4/0/1 NS ns-cloud-b1.googledomains.com., NS ns-cloud-b3.googledomains.com., NS ns-cloud-b4.googledomains.com., NS ns-cloud-b2.googledomains.com. (157)
15:14:12.107846 IP traefik.mydomain.com.42158 > one.one.one.one.domain: 52277+ A? ns-cloud-b1.googledomains.com. (47)
15:14:12.108006 IP traefik.mydomain.com.59332 > one.one.one.one.domain: 3646+ AAAA? ns-cloud-b1.googledomains.com. (47)
15:14:12.108708 IP one.one.one.one.domain > traefik.mydomain.com.42158: 52277 1/0/0 A 216.239.32.107 (63)
15:14:12.108868 IP one.one.one.one.domain > traefik.mydomain.com.59332: 3646 1/0/0 AAAA 2001:4860:4802:32::6b (75)
15:14:12.109297 IP traefik.mydomain.com.44789 > ns-cloud-b1.googledomains.com.domain: 52666 [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:12.110104 IP ns-cloud-b1.googledomains.com.domain > traefik.mydomain.com.44789: 52666 Refused 0/0/0 (41)
15:14:17.110914 IP traefik.mydomain.com.50836 > one.one.one.one.domain: 19355+ [1au] TXT? _acme-challenge.mydomain.com. (52)
15:14:17.113157 IP one.one.one.one.domain > traefik.mydomain.com.50836: 19355 4/0/1 TXT "99Y_CmSXEySi3cjUveKw-EtDzvS2HN_hRXQKasWImUE", TXT "iHo0X_Z6vGBlYVzUBKMYOYjp7DPoyu7-nMsLyFC-d9o", TXT "oCxdOrEnGoP6SevZ6sbw9_JRPFtJBYARcZRi7-OCOzQ", TXT "EH_b7fr1pbA5W9toUv8OXc-bpZZn1wE92ajtQDiNS6k" (276)

It seems that Google nameservers are responding with a "Refused".
Question is, why?

rg305 · March 22, 2022, 6:48pm

Obviously, Google nameservers don't think they are authoritative for that zone.
You need to ask 1.1.1.1 which nameservers it sees as authoritative for that zone.

onixid · March 22, 2022, 7:41pm

Hi @rg305, 1.1.1.1 and 8.8.8.8 both return the same NS:

# dig +short NS mydomain.com @1.1.1.1
ns-cloud-b1.googledomains.com.
ns-cloud-b3.googledomains.com.
ns-cloud-b4.googledomains.com.
ns-cloud-b2.googledomains.com.

# dig +short NS mydomain.com @8.8.8.8
ns-cloud-b1.googledomains.com.
ns-cloud-b3.googledomains.com.
ns-cloud-b4.googledomains.com.
ns-cloud-b2.googledomains.com.

But looking at the pcap file it seems that the problem is not with those 2 DNS servers, they do reply correctly to the acme challenge, what goes wrong is the final check on ns-cloud-b1.googledomains.com cause the "Refused" you see there comes from the nameserver and inspecting the response, I can see the response flags set as "Authoritative: Server is not an authority for domain".
So it seems like the NS doesn't consider itself as authoritative... but it is? I'm confused...