Getting manual cert fails

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: netreg.ahc.ufl.edu

I ran this command:certbot certonly --manual --manual-auth-hook '/etc/letsencrypt/acme-dns-auth.py' --preferred-challenges dns --debug-challenges -d netreg.ahc.ufl.edu

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for netreg.ahc.ufl.edu

Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.

Press Enter to Continue

Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
Domain: netreg.ahc.ufl.edu
Type: dns
Detail: During secondary validation: DNS problem: query timed out looking up TXT for _acme-challenge.netreg.ahc.ufl.edu

Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):

The operating system my web server runs on is (include version): Linux RHv8

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):certbot 1.22.0

2

Other info: have confirmed authoritative nameservers are resolving correctly internally and externally. The acme-dns server is ns.acme-challenge.ufhealth.org and it is resolving correctly. Can also "see" the TXT record in the acme-dns SQL database.

3

There's something weird with the configuration of those DNS servers, a lot of the hosts are in private IPv4 space. If in fact the data is synchronized between the public-IP hosts and the private-IP hosts, then I think it might be able to work eventually but take longer. But seems weird to me. Don't have time to dig into it more right now, but maybe you can explain more about how these DNS servers are set up and why it's like that?

4 Likes
4

Our authoritative enterprise nameservers have 3 DNS views, but almost all of our servers are on internal address space (internal view). We have the CNAME defined in the internal and external views (confirmed). The netreg.ahc.ufl.edu server on private address space can wget from acme-v02.api.letsencrypt.org via Squid http-based proxy (confirmed). Are there any other sites private server needs to access?

5

Just read your external resolution link. Unsure why ns3-ns6 are not aware of the delegated stub zone for acme-challenge.ufhealth.org. Am now reaching out to my enterprise IPAM folks.

6

I don't have time to dig through everything now, like I said, but the key idea of using acme-dns is that all outside-your-network DNS requests worldwide need to be able to follow the CNAME and get the TXT record that was put there, regardless of which authoritative DNS server they connect to.

2 Likes
7

Understood, but every Google/Cloudflare, etc, nameserver I test is able to resolve the TXT record for _acme-challenge.netreg.ahc.ufl.edu. Not seeing an issue doing that.

8

Not sure if those private IP addresses for ns{3..6} are that relevant: when doing a dig +trace it seems the "upstream" nameserver ens.name.ufl.edu only replies with ns{1..2} which have public IP addresses. Only when asking ns1.shands.ufl.edu or ns2.shands.ufl.edu directly for the authorative NS, it comes up with ns{1..6}. But such a request is not used in a +trace. Not sure if UnBound would handle this differently though.

Anyway, when I do dig +trace f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org. TXT, I'm getting some timeouts:

osiris@erazer ~ $ dig +trace f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org. TXT

; <<>> DiG 9.18.29 <<>> +trace f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org. TXT
;; global options: +cmd
.			45614	IN	NS	h.root-servers.net.
.			45614	IN	NS	m.root-servers.net.
.			45614	IN	NS	a.root-servers.net.
.			45614	IN	NS	g.root-servers.net.
.			45614	IN	NS	c.root-servers.net.
.			45614	IN	NS	e.root-servers.net.
.			45614	IN	NS	j.root-servers.net.
.			45614	IN	NS	k.root-servers.net.
.			45614	IN	NS	b.root-servers.net.
.			45614	IN	NS	d.root-servers.net.
.			45614	IN	NS	f.root-servers.net.
.			45614	IN	NS	l.root-servers.net.
.			45614	IN	NS	i.root-servers.net.
.			45614	IN	RRSIG	NS 8 0 518400 20250628170000 20250615160000 53148 . mprGuQ4WJtjtiHYYBcnlnXFjlJbor/Du5J/pTWUxyHXxARIqZjHlHRJh tJB/XWQ/UdL0nGkufNL1Zl1XHrHvWM5Ks8bpDX/94adyQ0lGe8olrQ3V QjLoDk/XbXmNYhA4/jyX/J3zgBsFFhIMYedTfcfvaim86ay+orlNSs8o NgeCfv9wKGGVjy4/QYImKlxUJkVfiwAvYkM3HTp3tdaoqixjoWQx8zRc Ddi11ke+o1wKzXpzjB8TDQg0WESKE/tvhBUHdSFWAk24usAq7Es4uad9 Vu/SuBJKVHZkE92RVDtFuIK/KBjSeGA7KWYJtnwkTE0AujuU0+EaxwCX a9qmmA==
;; Received 525 bytes from 185.93.175.43#53(185.93.175.43) in 11 ms

org.			172800	IN	NS	b2.org.afilias-nst.org.
org.			172800	IN	NS	b0.org.afilias-nst.org.
org.			172800	IN	NS	a0.org.afilias-nst.info.
org.			172800	IN	NS	a2.org.afilias-nst.info.
org.			172800	IN	NS	d0.org.afilias-nst.org.
org.			172800	IN	NS	c0.org.afilias-nst.info.
org.			86400	IN	DS	26974 8 2 4FEDE294C53F438A158C41D39489CD78A86BEB0D8A0AEAFF14745C0D 16E1DE32
org.			86400	IN	RRSIG	DS 8 1 86400 20250629050000 20250616040000 53148 . pAUgg2GREJQ+dKbfaqqC6geLU9A+6j3I7aeVypTbyfMYSH4ulSZ0AiGu qI4QHO49JmH6ZUV3Dt/rqbm+0FQpjvGNerhnJ5Nno0GBy0xc6Ow7Z0Ik xiNx6Pl2kaphjN7Z7mGamAxDv52evWvBIA/UKDUK4aPKgqL3N8WDeivC uvkgPIc2LL3rRu8gH+CNLyKZV9yP2cxdnB8gycXw0fUHNsZts5n3rZDV V8vCJCQlgOLkI/OZrv+OIFBve7f6niw4+9K98zRzLyXVi19OG8UiqiUp A1XlNlQnBFBI3dnJve+QdSfuV9PiU7UpOFEPZtsZhwW6jTmC7h15UlJi 1IhQcw==
;; Received 864 bytes from 2001:500:12::d0d#53(g.root-servers.net) in 25 ms

;; communications error to 2001:500:48::1#53: timed out
;; communications error to 2001:500:48::1#53: timed out
;; communications error to 2001:500:48::1#53: timed out
ufhealth.org.		3600	IN	NS	ns2.shands.ufl.edu.
ufhealth.org.		3600	IN	NS	ns1.shands.ufl.edu.
gdtpongmpok61u9lvnipqor8lra9l4t0.org. 3600 IN NSEC3 1 1 0 332539EE7F95C32A GDTV35RCVC3A33DQ9ALQ59QE9QJIB3KU NS SOA RRSIG DNSKEY NSEC3PARAM
gdtpongmpok61u9lvnipqor8lra9l4t0.org. 3600 IN RRSIG NSEC3 8 2 3600 20250707161312 20250616151312 321 org. ZXVP9w1gM/9eLzb8XeCzE3MnCWM3N3QklbXmXU9VA57DSe1YUEgTWD/u 2GFIQAvgf2wP+ffpQr59O3p/B88JB5PCKY4Ofrw31VuPtjqEz7ioDHEX C/7yjblQofvTjGTs6JPSOm3XaJhTlwzg/K08dB3ynAxJmmYx4BvSa/4F LLg=
luiid1rkvria4b4dht4fu0385vuolhmm.org. 3600 IN NSEC3 1 1 0 332539EE7F95C32A LUIK9IRSLKQC8T8FHTER7P9L33KEAD2A NS DS RRSIG
luiid1rkvria4b4dht4fu0385vuolhmm.org. 3600 IN RRSIG NSEC3 8 2 3600 20250630152615 20250609142615 321 org. Mbrp1M9wGvuPO4/ZjTWAtuVJJ4TPch3eRkAMT1++TaYDmx4MbHtCliQi 8yRSJ6Pce1oEoBpcOcRWWrTF3oupk5O3GHj3i6rqYP1M4jEVZW8TVfxu 6XN9T9dLL85JR22uyHsbNGx4M80h5bX52do5TEHyKpRRMyby2Mk/MxpL Xbc=
;; Received 644 bytes from 199.19.53.1#53(c0.org.afilias-nst.info) in 231 ms

acme-challenge.ufhealth.org. 10800 IN	NS	acme-challenge.ufhealth.org.
couldn't get address for 'acme-challenge.ufhealth.org': failure
dig: couldn't get address for 'acme-challenge.ufhealth.org': no more
osiris@erazer ~ $

Although I think those timeouts are from the .org nameservers from "afilias-nst".

That said, for some reason my dig is suddenly not able to continue. The acme-challenge.ufhealth.org. 10800 IN NS acme-challenge.ufhealth.org. part without a glue record is obviously a catch22 and prevents resolving it properly.

That said, UnBoundTest at https://unboundtest.com/m/TXT/f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org/FM3WCGNE doesn't seem to have any issue..?

Hm, weird.. ns1.shands.ufl.edu. and ns2.shands.ufl.edu. do seem to provide the necessary glue:

osiris@erazer ~ $ dig @ns1.shands.ufl.edu. acme-challenge.ufhealth.org.

; <<>> DiG 9.18.29 <<>> @ns1.shands.ufl.edu. acme-challenge.ufhealth.org.
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46560
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 5ec5cfcdad92c36901000000685046234745fc988c1ecf85 (good)
;; QUESTION SECTION:
;acme-challenge.ufhealth.org.	IN	A

;; AUTHORITY SECTION:
acme-challenge.ufhealth.org. 10800 IN	NS	acme-challenge.ufhealth.org.

;; ADDITIONAL SECTION:
acme-challenge.ufhealth.org. 10800 IN	A	159.178.62.70

;; Query time: 164 msec
;; SERVER: 159.178.61.125#53(ns1.shands.ufl.edu.) (UDP)
;; WHEN: Mon Jun 16 18:28:09 CEST 2025
;; MSG SIZE  rcvd: 114

osiris@erazer ~ $ dig @ns2.shands.ufl.edu. acme-challenge.ufhealth.org.

; <<>> DiG 9.18.29 <<>> @ns2.shands.ufl.edu. acme-challenge.ufhealth.org.
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1329
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
; COOKIE: 2b24fa6ad6c6e810010000006850462bd26309648f176213 (good)
;; QUESTION SECTION:
;acme-challenge.ufhealth.org.	IN	A

;; AUTHORITY SECTION:
acme-challenge.ufhealth.org. 10800 IN	NS	acme-challenge.ufhealth.org.

;; ADDITIONAL SECTION:
acme-challenge.ufhealth.org. 10800 IN	A	159.178.62.70

;; Query time: 166 msec
;; SERVER: 159.178.81.50#53(ns2.shands.ufl.edu.) (UDP)
;; WHEN: Mon Jun 16 18:28:17 CEST 2025
;; MSG SIZE  rcvd: 114

osiris@erazer ~ $

However, I'm not able to connect to this nameserver 159.178.62.70:

osiris@erazer ~ $ dig @159.178.62.70 f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org. TXT
;; communications error to 159.178.62.70#53: timed out
;; communications error to 159.178.62.70#53: timed out
;; communications error to 159.178.62.70#53: timed out

; <<>> DiG 9.18.29 <<>> @159.178.62.70 f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org. TXT
; (1 server found)
;; global options: +cmd
;; no servers could be reached
osiris@erazer ~ $

Is there geoblocking going on for 159.178.62.70 perhaps?

Also, DNSViz reports a NXDOMAIN for f6355b7a-8634-49ac-b397-b26e2c71a121.acme-challenge.ufhealth.org.? Which is in contrast with the TXT result from UnBoundTest.. :thinking:

3 Likes
9

That appears to be the problem, though I don't understand why. If you're searching externally, it should be non-recursive: would reach our enterprise nameservers who would provide 159.178.62.70 for you to query.

10

See e.g. DNS Checker - DNS Check Propagation Tool.

It's having trouble from almost everywhere in the world, except most of the US (1 fails) and apparently Ireland and 1 out of 2 from Australia..

Note that LE too performs the challenge from multiple vantage points in the world. The "During secondary validation" points out that one or more of those non-primary validation points failed, probably a non-US one.

5 Likes
11

That's the issue. Feel sure my firewall folks enabled a geolocation policy different from our enterprise servers. Will follow up w/them further to determine how we can enable it to be queried freely. Thanks, much, for helping me troubleshoot this.

4 Likes