Can't renew with dns-01: Waiting for DNS record propagation

pjones · August 12, 2022, 3:25pm

I ran this command:

lego --accept-tos --path . -d stats.devalot.com --email domains@pmade.com --key-type ec256 --dns cloudflare renew --days 30

It produced this output:

2022/08/12 09:50:47 [INFO] [stats.devalot.com] acme: Trying renewal with 659 hours remaining
2022/08/12 09:50:47 [INFO] [stats.devalot.com] acme: Obtaining bundled SAN certificate
2022/08/12 09:50:48 [INFO] [stats.devalot.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/141175587277
2022/08/12 09:50:48 [INFO] [stats.devalot.com] acme: Could not find solver for: tls-alpn-01
2022/08/12 09:50:48 [INFO] [stats.devalot.com] acme: Could not find solver for: http-01
2022/08/12 09:50:48 [INFO] [stats.devalot.com] acme: use dns-01 solver
2022/08/12 09:50:48 [INFO] [stats.devalot.com] acme: Preparing to solve DNS-01
2022/08/12 09:50:48 [INFO] cloudflare: new record for stats.devalot.com, ID XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
2022/08/12 09:50:48 [INFO] [stats.devalot.com] acme: Trying to solve DNS-01
2022/08/12 09:50:48 [INFO] [stats.devalot.com] acme: Checking DNS record propagation using [10.0.1.1:53 [2600:8800:1700:242:2e0:edff:fecf:20ba]:53]
2022/08/12 09:50:50 [INFO] Wait for propagation [timeout: 5m0s, interval: 2s]
2022/08/12 09:51:01 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:51:13 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:51:25 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:51:37 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:51:49 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:52:01 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:52:13 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:52:25 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:52:37 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:52:49 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:53:01 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:53:13 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:53:25 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:53:37 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:53:49 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:54:01 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:54:13 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:54:25 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:54:37 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:54:49 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:55:01 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:55:13 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:55:25 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:55:37 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:55:49 [INFO] [stats.devalot.com] acme: Waiting for DNS record propagation.
2022/08/12 09:55:51 [INFO] [stats.devalot.com] acme: Cleaning DNS-01 challenge
2022/08/12 09:55:51 [INFO] retry due to: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/authz-v3/141175587277 :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: "0102fivPnYZT9Unme6mC2__beuDiEUL7770-xtW>
2022/08/12 09:55:52 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/141175587277
2022/08/12 09:55:52 error: one or more domains had a problem:
[stats.devalot.com] time limit exceeded: last error: read udp [2600:8800:1700:242::2000]:34999->[2606:4700:58::a29f:2cd1]:53: i/o timeout

The operating system:

NixOS 22.05 (Linux 5.15)

I can login to a root shell on my machine (yes or no, or I don't know): Yes.

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

lego version 4.6.0 linux/amd64

I've been using dns-01 for quite some time so I'm not sure why this renewal isn't working.

Over the last week I have tried everything I know to debug this but I don't feel I'm any closer to understanding what's going on. I've let the process run for an hour yesterday.

Each time I run lego I immediately see the TXT records appear in the Cloudflare dashboard. And within a very short period of time I see the record propagated out to various DNS servers across the internet and the resolver on the machine running lego can also see the TXT record.

Things I've tried:

Disabling my outbound firewall rules (pfBlockerNG)
Giving lego various --dns.resolvers to try
Running dig on the same machine as lego to verify I see the _acme-challenge TXT record
Trying various timeout/poll values for Cloudflare and --dns-timeout
Let's Debug

At this point I've exhausted my knowledge of DNS and lego and would appreciate some help.

Thank you.

Osiris · August 12, 2022, 4:10pm

Are those DNS servers even working? It seems lego will check the propogation using server side configured DNS servers first before triggering the validation at the ACME server.

You can clearly see it tries to check this propogation for the 5 minutes which is configured for this, after which it stops. So to me, this sounds like your lego client tries the above listed DNS servers, but after 5 minutes of trying, it just time outs, suggesting to me the DNS servers are not working.

pjones · August 12, 2022, 4:15pm

Osiris: I've tried various DNS servers via --dns.resolvers and while lego is running, on the same machine I can use dig to fetch the TXT records from the DNS server lego is configured to use.

Any idea why dig works but lego doesn't?

Osiris · August 12, 2022, 4:19pm

I have absolutely no idea what so ever. I've never used lego and I'm not familiar with it. Perhaps the lego devs might know more.

pjones · August 12, 2022, 4:20pm

Thank you for trying to help. I appreciate it.

I wish that lego was a bit more verbose. When it reports that it's "Waiting for DNS record propagation" it's not clear if the DNS server failed to respond, or responded with a "record not found" error.

MikeMcQ · August 12, 2022, 4:29pm

I think asking the devs on github is good idea too.

Did you try running with below option? Sounds like it disables the pre-check lego does. Might help diagnose where problem is

--dns.disable-cp By setting this flag to true, disables the need to wait the propagation of the TXT record to all authoritative name servers. (default: false)
https://go-acme.github.io/lego/usage/cli/options/

pjones · August 12, 2022, 4:40pm

@MikeMcQ Yes, I did try using that option.

I think I just figured it out. I had been ignoring the final line in the lego output:

[stats.devalot.com] time limit exceeded: last error: read udp [2600:8800:1700:242::2000]:34999->[2606:4700:58::a29f:2cd1]:53: i/o timeout

Even if I give lego an IPv4 address to the --dns.resolvers flag, it is still trying to connect to an IPv6 address. According to dig -x that IPv6 address is the root name server for my domain (cloudflare). If I use dig to fetch records over IPv6 if fails! @Osiris was correct, the DNS server wasn't responding!

I disabled IPv6 on the machine running lego and the certificate was able to renew!

I have no idea why DNS on IPv6 isn't working. I just tried to make an HTTP request over IPv6 and it worked. I guess I have something else to fix now.

Thanks for all the help!

Osiris · August 12, 2022, 4:51pm

Sounds like a bug in lego to me! Why would it try all kinds of IP addresses you didn't enter on the CLI? Or do you have those DNS servers configured in a lego configuration file perhaps, which is additive to the --dns.resolvers flag or something?

pjones · August 12, 2022, 10:31pm

From what I've been able to piece together, lego doesn't use the --dns.resolvers flag when doing dns-01. That makes sense because the LE server needs to read the TXT record directly off of the authoritative name server.

In my case, the authoritative name server has multiple IP address, half of which are IPv6. When lego tries to query the server for the TXT record, it appears that even though the response times out, it doesn't log that fact nor does it try a different address.

I added a comment on an open issue that is slightly related to this.

MikeMcQ · August 12, 2022, 10:42pm

Thanks for the update.

I'm a little surprised the --dns.disable-cp did not work-around the problem like it did for the others in that thread

pjones · August 13, 2022, 4:34pm

I'm surprised as well. But I did try it and lego failed right away.

anusree · August 16, 2022, 4:16pm

DNS propagation is the time frame it takes for DNS changes to be updated across the Internet. A change to a DNS record—for example, changing the IP address defined for a specific hostname—can take up to 72 hours to propagate worldwide, although it typically takes a few hours.

rmbolger · August 16, 2022, 5:30pm

For the purposes of certificate validation, the record changes don't need to propagate across the entire Internet (accounting for TTL and cache expirations). The validation servers directly query the authoritative nameservers for the domain as @pjones correctly mentioned (the servers pointed to by the NS records). So those authoritative nameservers are the only thing the changes need to propagate to which is usually on the order of seconds or minutes depending on the underlying DNS software/architecture. Though there are some providers that take much longer.

This gets additionally complex for large providers who may be using anycast where a single nameserver IP address might be served by multiple machines across the globe. So the answer you get may depend on where you are querying the name from. Though for a provider like Cloudflare, that global anycast propagation still seems to happen in roughly 15 seconds after a record change in my experience.

system · September 15, 2022, 5:30pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.