Cerbot renewal error - No TXT record found

[karnikram]

My domain is: rrc.iiit.ac.in (within local network). We have a DNAME on our local DNS to point *.rrc.iiit.ac.in to *.rrcx.tk. rrcx.tk is our global domain managed through cloudflare where we insert all our A, CNAME, TXT records.

I ran this command: certbot certonly -d "*.rrc.iiit.ac.in" --manual on our server inside our local network.

It produced this output:

Please deploy a DNS TXT record under the name
_acme-challenge.rrc.iiit.ac.in with the following value:

sOWBirCDa2cu01mXHzRxStlFb01MqkApCwSU31744ZY

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. rrc.iiit.ac.in (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: No TXT record found at _acme-challenge.rrc.iiit.ac.in

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: rrc.iiit.ac.in
   Type:   unauthorized
   Detail: No TXT record found at _acme-challenge.rrc.iiit.ac.in

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

The operating system my web server runs on is (include version): Ubuntu 18.04.4

I can login to a root shell on my machine (yes or no, or I don't know): Yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

I've created the TXT record and it clearly exists:
Ran on the same server,

$ nslookup -q=TXT _acme-challenge.rrc.iiit.ac.in
Server:         10.4.20.222
Address:        10.4.20.222#53

rrc.iiit.ac.in  dname = rrcx.tk.
_acme-challenge.rrc.iiit.ac.in  canonical name = _acme-challenge.rrcx.tk.
_acme-challenge.rrcx.tk text = "sOWBirCDa2cu01mXHzRxStlFb01MqkApCwSU31744ZY"

Ran outside our network,

$ nslookup -q=TXT _acme-challenge.rrcx.tk
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
_acme-challenge.rrcx.tk	text = "sOWBirCDa2cu01mXHzRxStlFb01MqkApCwSU31744ZY"

I'm not sure what is going wrong.

[_az]

@karnikram

Ran outside our network,

$ nslookup -q=TXT _acme-challenge.rrcx.tk

When I query up _acme-challenge.rrc.iiit.ac.in, I get a CNAME to _acme-challenge.rrcx.ml.

.ml. Not .tk as in your nslookup command.

There's no TXT record at .ml.

[karnikram]

Thank you for your reply @_az

This is strange. rrcx.ml was our old domain which was in use only until yesterday. rrcx.tk is our new domain and we've updated our DNAME to point *.rrc.iiit.ac.in to *.rrcx.tk (instead of *.rrcx.ml)

From inside our network it seems to work. But like you've pointed out it appears that outside our network it is still being mapped to the old domain. Could this be because the other nameservers (like 8.8.8.8) are using cached results?

[_az]

@karnikram

Could this be because the other nameservers (like 8.8.8.8) are using cached results?

I don't think so, querying your nameservers directly produces that result:

$ dig +noall +answer @ns.iiit.ac.in _acme-challenge.rrc.iiit.ac.in
rrc.iiit.ac.in.         120     IN      DNAME   rrcx.ml.
_acme-challenge.rrc.iiit.ac.in. 120 IN  CNAME   _acme-challenge.rrcx.ml.

[griffin]

I concur.

[karnikram]

Thank you for your help @_az, @griffin.

The DNAME in our public facing ns weren't updated. It is working now. It makes sense for certbot to do the check from outside the network.

[rg305]

@karnikram

$ nslookup -q=TXT _acme-challenge.rrc.iiit.ac.in
Server:         10.4.20.222
Address:        10.4.20.222#53

The world can't see your Internal DNS server.

[karnikram]

True. But the world can't see any of our sites either They're only on our local network.

[rg305]

OK, but you want a world trusted cert.
So you need to place your TXT records in a DNS that the world can reach.

[_az]

From their original post, it seems that they have an internal DNS on 10/8, and an external view hosted by Cloudflare. So everything looks fixed now.

[Osiris]

@_az

So everything looks fixed now.

I concur, there's a newly issued certificate visible from today.

@karnikram managed to get the DNAME/CNAME for the challenge RR in place on the external DNS server:

rrc.iiit.ac.in.		120	IN	DNAME	rrcx.tk.
_acme-challenge.rrc.iiit.ac.in.	120 IN	CNAME	_acme-challenge.rrcx.tk.
;; Received 110 bytes from 14.139.82.4#53(ns2.iiit.ac.in) in 410 ms

However, while I'm not that experienced with DNAME RRs, isn't the above a little bit redundant? As far as I know, the DNAME RR would do the exact same thing as the specific CNAME RR?

[rg305]

@Osiris

the DNAME RR would do the exact same thing as the specific CNAME RR

Yes, and a lot more too.

[karnikram]

Yes, the orginal issue is fixed now. Sorry, I think I wasn't clear in my previous reply when I said it's working.

@Osiris @rg305 I noticed that too. I haven't inserted such a CNAME on the rrcx.tk cloudflare DNS. And only the DNAME exisits on our own DNS servers. I'm not sure how it appeared. Maybe it is auto generated as part of the resolution process. (I'm just a grad student managing our small lab's servers so please bear with my naivety : ) )

[griffin]

Just for everyone's lucidity...

Since every name modified by a DNAME could also be expressed as a single CNAME the server will return both the DNAME RR and a synthesized CNAME version. Since all resolvers are CNAME aware (CNAME dates from the earliest years of DNS - somewhere around the time dinosaurs walked the earth) but not all resolvers are DNAME aware, all resolvers will find something they like (DNAME or CNAME or both even) in the response.

https://www.zytrax.com/books/dns/ch8/dname.html