DNS provider change - validation uses the old DNS provider

I would like to ask for your help.

We have just changed the DNS provider of one of our domains: stg.containers.appdomain.cloud. We create subdomains under this domain and we create certificates for those subdomains. We use the LEGO client for that.

The change happened around 15:15 UTC. The old NS records in the old DNS provider had TTL 300 sec. But it seems, Let's Encrypt still looks for the TXT records for validation in our old DNS servers.

The LEGO client manages the TXT records in the new DNS servers, we checked those.

The old DNS servers were:

plato.ns.cloudflare.com
maria.ns.cloudflare.com

New DNS servers:

a1-180.akam.net.
a22-66.akam.net.
a10-67.akam.net.
a8-66.akam.net.
a5-67.akam.net.
a9-66.akam.net.

We keep getting errors like

"NS maria.ns.cloudflare.com. returned NXDOMAIN"
"NS maria.ns.cloudflare.com. did not return the expected TXT record"

Could you please help in understanding why Let's Encrypt does not follow the new DNS servers when validating the TXT records?

nslookup for SOA:

nslookup

set query=soa
my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud

Authoritative answers can be found from:
stg.containers.appdomain.cloud
origin = a1-180.akam.net
mail addr = hostmaster.stg.containers.appdomain.cloud
serial = 2021076100
refresh = 3600
retry = 600
expire = 604800
minimum = 300

nslookup for NS:

nslookup

set query=ns
my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud

Authoritative answers can be found from:
stg.containers.appdomain.cloud
origin = a1-180.akam.net
mail addr = hostmaster.stg.containers.appdomain.cloud
serial = 2021075874
refresh = 3600
retry = 600
expire = 604800
minimum = 300

Thank you!

My domain is: a test domain: my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud

I ran this command: I use the LEGO client to obtain certificates with DNS-01

It produced this output: N/A

My web server is (include version): N/A

The operating system my web server runs on is (include version): N/A

My hosting provider, if applicable, is: We changed from Cloudflare to Akamai

I can login to a root shell on my machine (yes or no, or I don't know): N/A

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): N/A

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): N/A

2 Likes

If you do a dig +trace you still see the Cloudflare NS servers between the ns119.name.cloud.ibm.com like nameservers and your akam.net nameservers. Is that intentional?

Also, the errors you've provided don't actually look like errors returned by the Let's Encrypt validation server. Perhaps those are local tests by your ACME client?

3 Likes

Thank you for the fast answer!

If you do a dig +trace you still see the Cloudflare NS servers between the ns119.name.cloud.ibm.com like nameservers and your akam.net nameservers. Is that intentional?

I can see this. For me all the nameservers that serve the stg.containers.appdomain.cloud are from the new DNS provider.

➜ dig +trace my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud

; <<>> DiG 9.10.6 <<>> +trace my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud
;; global options: +cmd
.			61676	IN	NS	b.root-servers.net.
.			61676	IN	NS	k.root-servers.net.
.			61676	IN	NS	h.root-servers.net.
.			61676	IN	NS	l.root-servers.net.
.			61676	IN	NS	e.root-servers.net.
.			61676	IN	NS	g.root-servers.net.
.			61676	IN	NS	m.root-servers.net.
.			61676	IN	NS	i.root-servers.net.
.			61676	IN	NS	j.root-servers.net.
.			61676	IN	NS	a.root-servers.net.
.			61676	IN	NS	d.root-servers.net.
.			61676	IN	NS	f.root-servers.net.
.			61676	IN	NS	c.root-servers.net.
.			61676	IN	RRSIG	NS 8 0 518400 20210701050000 20210618040000 14631 . Fwl2FI0x2JggipKawV0H/O9ftd1Ae84G0Od2tY4Kvtfc1KrIXmnzfS33 1isti6bmJGsW8AOqS7T7eyrOMxADq99tIVg5+8YoJN/O6kRMYWchxyAf v76A67ibjnMqynG991onwUBWzvdseaJTlZwBlKVIpDaxdfKF9QRc55A2 qQoK3vVbOwkfBQsv1aOXJVtgB4veM0okwr5jpH8OEYpXQmAUz6ZIrYMe aupQ31A4RQRp7O8q0LE4+0kNFBNVmTiVSaCobIQpqZ9bhZt9HmocnqV0 I1Fi51lQ5zzI0LDrk+HKGbDhG5B1jQVnggyuPlM7zVa721O/9RPrXoKM d5eybA==
;; Received 525 bytes from 213.46.246.53#53(213.46.246.53) in 43 ms

cloud.			172800	IN	NS	a.nic.cloud.
cloud.			172800	IN	NS	d.nic.cloud.
cloud.			172800	IN	NS	c.nic.cloud.
cloud.			172800	IN	NS	b.nic.cloud.
cloud.			86400	IN	DS	49804 8 1 04B537D00C915E782432584948550BBB39FF88A0
cloud.			86400	IN	DS	49804 8 2 7781C5D4ECF4BD0845B49D29B2E79BD1DA1865725096FAEB2D0393B7 8FD1049C
cloud.			86400	IN	RRSIG	DS 8 1 86400 20210701170000 20210618160000 14631 . ZE8S1F7HzI8xngauHQv/bReaDRcNSA/dP76LN9VzdLwdcou4t/C1zc1d 56mmYBypFsyXhB+K+wOoGFOuHM73+rYKWkT2KIhO53k5D/PJEiIs4Cb4 oXYaUWSYSbBePf6CKv+rukBZjxd3w0HEflcnS7Xx4qByHjUzV1kCSw3/ mNkAM5wN+RB69HLZYFmpY91TpkZgTqf85gp2Anr+2/kcgOkcXKdYAEG9 8QitrfC4iOwE/Y1fi4G4kDXKlVIegAtz3vEoV7hgxAfyrvVoWsC7Z0Lh mlRQgmOkZ9ihr93xd4VMaXvvgC77KhdnObqt30Q0pwfJhU4FnIGSiXu1 MrWstA==
;; Received 749 bytes from 192.36.148.17#53(i.root-servers.net) in 38 ms

appdomain.cloud.	3600	IN	NS	ns130.name.cloud.ibm.com.
appdomain.cloud.	3600	IN	NS	ns119.name.cloud.ibm.com.
k6nnfqiccqt79kvb1rieurknvn2tfr6a.cloud.	1800 IN	NSEC3 1 1 1 AA821FC0 K73185KG2C12RJIEF8OCETL5OBDHKBMO  NS SOA RRSIG DNSKEY NSEC3PARAM
k6nnfqiccqt79kvb1rieurknvn2tfr6a.cloud.	1800 IN	RRSIG NSEC3 8 2 1800 20210716105856 20210616100218 2538 cloud. SvRwtSXvjPbervUhBaNghFf+Gf/r8gnvZwjYSwv/sNez/OCdGeq8DOh3 kpG6dEJjIFaXqThdoMQkJvjFGurEutFSzxyLQOC9cnid/wuSkHwRtDeN tK5qPk+TNZ+mvRoHycCPexjwB8YYlA753ELzVI2bnAjGVb7rpbgKCUWC vpng/R4N+7ll13Pr/rp2NLObHfKkuBO6z93MF0k0EXeiiw==
i6ba6kksu7pa8bejtfqtoor9ebdq5s95.cloud.	1800 IN	NSEC3 1 1 1 AA821FC0 I6G0IFSSIRLNS8DC4U243HNGK6MJQJSB  NS DS RRSIG
i6ba6kksu7pa8bejtfqtoor9ebdq5s95.cloud.	1800 IN	RRSIG NSEC3 8 2 1800 20210712052737 20210612052330 2538 cloud. DfuS2jUTVD8CNByL9tZCYoJD3DGgLEpfgf1Nds9ZZskqr7jCo8LW1vaq FPCCJmVOpaxpcr987G8/22bSsFBi0p8o8ozR7dXcQUYfzkbkVf6E3Pg8 mufG7YaEzn+b2Rfx5sc058TZDfi2CuOFH2XfFSC7Q6g2DlduUfAwQcRo X/J4oYu0z9XO5Lvuoc7T7Q/jBCZyDC5OlGaKKOsp32yAMw==
;; Received 758 bytes from 37.209.196.10#53(c.nic.cloud) in 43 ms

containers.appdomain.cloud. 900	IN	NS	jake.ns.cloudflare.com.
containers.appdomain.cloud. 900	IN	NS	ada.ns.cloudflare.com.
;; Received 178 bytes from 162.159.42.65#53(ns130.name.cloud.ibm.com) in 26 ms

stg.containers.appdomain.cloud.	300 IN	NS	a1-180.akam.net.
stg.containers.appdomain.cloud.	300 IN	NS	a22-66.akam.net.
stg.containers.appdomain.cloud.	300 IN	NS	a10-67.akam.net.
stg.containers.appdomain.cloud.	300 IN	NS	a8-66.akam.net.
stg.containers.appdomain.cloud.	300 IN	NS	a5-67.akam.net.
stg.containers.appdomain.cloud.	300 IN	NS	a9-66.akam.net.
;; Received 255 bytes from 108.162.192.54#53(ada.ns.cloudflare.com) in 33 ms

my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud. 60 IN A 169.60.227.34
;; Received 140 bytes from 2.16.40.66#53(a8-66.akam.net) in 41 ms

Also, the errors you've provided don't actually look like errors returned by the Let's Encrypt validation server. Perhaps those are local tests by your ACME client?

We have to check the client then. Thank you!

3 Likes

In the end, yes, but isn't it a little bit weird to find two Cloudflare nameservers in between? Or is that intentional? Probably not the issue you're looking at now, by the way.

By the way, you can use the site unboundtest.com to see how Let's Encrypt would resolve a certain RR:

https://unboundtest.com/m/TXT/_acme-challenge.my-test-cluster-2-7a13fae41c466d2224b963f00c9f778f-0000.us-south.stg.containers.appdomain.cloud/GX6IH7EQ

Looks fine to me with akam.net at the end of the resolving path.

I'm guessing the CNAME is also intentional?

3 Likes

In the end, yes, but isn't it a little bit weird to find two Cloudflare nameservers in between? Or is that intentional?

Yes, this is how we want this. We keep containers.appdomain.cloud in the old DNS for the time being and we move the stg.containers.appdomain.cloud to the new DNS only.

I'm guessing the CNAME is also intentional?

No, that should not be like that. Even more strange I cannot see that CNAME when checking it via the DNS provider's UI. Investigating that, too.

By the way, you can use the site unboundtest.com to see how Let's Encrypt would resolve a certain RR

Thank you for the hint! I am sure it will be very useful for us.

4 Likes

No, that should not be like that. Even more strange I cannot see that CNAME when checking it via the DNS provider's UI. Investigating that, too.

The CNAME is returned because we register a CNAME for the wildcard subdomain. As the _acme-challenge subdomain is not in the DNS at this point of time (we have a pause between Let's Encrypt registraion attempts), it matches the wildcard domain and returns with the CNAME record.

2 Likes

You might want to keep a TXT record around temporarily for debugging purposes.

2 Likes

Thank you for your help!
The result of our debugging is, that an intermediate cache kept the old DNS NS record for too long time - it was the reason for those error messages, that actually come from the LEGO client we use.

I am very thankful for your help! Excuse me for the false alert.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.