DNS-01 challenge fails

My domain is:
domesweetdome.us.com

I ran this command:
certbot certonly --dry-run --debug-challenges --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/james/rfc2136.ini --dns-rfc2136-propagation-seconds 5 --preferred-challenges=dns --email redacted@Idontwantbotspam --agree-tos -d domesweetdome.us.com -d www.domesweetdome.us.com -d ftp.domesweetdome.us.com -d mail.domesweetdome.us.com -d smtp.domesweetdome.us.com -d imap.domesweetdome.us.com -d pop3.domesweetdome.us.com

It produced this output:
Saving debug log to /var/log/certbot/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for domesweetdome.us.com
dns-01 challenge for ftp.domesweetdome.us.com
dns-01 challenge for imap.domesweetdome.us.com
dns-01 challenge for mail.domesweetdome.us.com
dns-01 challenge for pop3.domesweetdome.us.com
dns-01 challenge for smtp.domesweetdome.us.com
dns-01 challenge for www.domesweetdome.us.com
Waiting 5 seconds for DNS changes to propagate
Waiting for verification…


Challenges loaded. Press continue to submit to CA. Pass “-v” for more info about
challenges.

Press Enter to Continue
Cleaning up challenges
Failed authorization procedure. imap.domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.imap.domesweetdome.us.com, pop3.domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.pop3.domesweetdome.us.com, mail.domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.mail.domesweetdome.us.com, www.domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.domesweetdome.us.com, domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.domesweetdome.us.com, smtp.domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.smtp.domesweetdome.us.com, ftp.domesweetdome.us.com (dns-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.ftp.domesweetdome.us.com

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: imap.domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.imap.domesweetdome.us.com

    Domain: pop3.domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.pop3.domesweetdome.us.com

    Domain: mail.domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.mail.domesweetdome.us.com

    Domain: www.domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.www.domesweetdome.us.com

    Domain: domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.domesweetdome.us.com

    Domain: smtp.domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.smtp.domesweetdome.us.com

    Domain: ftp.domesweetdome.us.com
    Type: None
    Detail: DNS problem: NXDOMAIN looking up TXT for
    _acme-challenge.ftp.domesweetdome.us.com

My web server is (include version):
Apache2 version 2.4.33
Bind version 9.11.2

The operating system my web server runs on is (include version):
OpenSuSE Leap 15.0

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.30.2

When I run certbot in a dry-run to test if I can get a certificate, at the point that is pauses for doing the -debug-challenges I cd into where named is running chrooted and sync’ed the journal using the command rndc sync I then looked at the conf file for domesweetdome.us.com and I can see that indeed the TXT records for the challenge are recorded. -

$ORIGIN .
$TTL 172800 ; 2 days
domesweetdome.us.com IN SOA ns1.domesweetdome.us.com. redacted.Idontwantbotspam.com (
2019031222 ; serial
10800 ; refresh (3 hours)
3600 ; retry (1 hour)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS ns1.domesweetdome.us.com.
A 205.151.255.18
MX 1 mail.domesweetdome.us.com.
TXT “v=spf1 a mx a:mail.domesweetdome.us.com mx:mail.domesweetdome.us.com ~all”
$ORIGIN domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “PBqho5qbmiSYzNAEkMbcTnwiPeohDKC3GZ8GXLc4ns4”
$TTL 172800 ; 2 days
_dmarc TXT "v=DMARC1;p=quarantine;sp=quarantine;pct=20;aspf=r;fo=0;ri=86400;rua=mailto:postmaster@domesweetdome.us.com"
ftp CNAME domesweetdome.us.com.
$ORIGIN ftp.domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “RHqJSoREDMeR5Z813dXSwKQNz2Ykz5T5hyvrGGD-zu0”
$ORIGIN domesweetdome.us.com.
$TTL 172800 ; 2 days
imap A 205.151.255.18
$ORIGIN imap.domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “npsYqsHZ3h9SUVpadI4VkZLPRqJ8jJSKpUm9rXQJcyU”
$ORIGIN domesweetdome.us.com.
$TTL 172800 ; 2 days
mail A 205.151.255.18
$ORIGIN mail.domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “6j3cfMVAnY1yq-K6xoWQbLTAM9YN-EkpIb5SFUPE1mk”
$ORIGIN domesweetdome.us.com.
$TTL 172800 ; 2 days
ns1 A 127.0.0.1
pop3 A 205.151.255.18
$ORIGIN pop3.domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “5HSLlb3bSu-iJOEwWLw9dScsbHjAC88tnNmzC7Kb7PY”
$ORIGIN domesweetdome.us.com.
$TTL 172800 ; 2 days
smtp A 205.151.255.18
$ORIGIN smtp.domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “ay53r9suUZCMUqp-UeTR19207PAFcW1RAEdVjaURA8g”
$ORIGIN domesweetdome.us.com.
$TTL 172800 ; 2 days
test A 205.151.255.18
www CNAME domesweetdome.us.com.
$ORIGIN www.domesweetdome.us.com.
$TTL 120 ; 2 minutes
_acme-challenge TXT “9SgyBSepVasWRB-f99eiBm0q1pB8nHrehMLkItCMkBM”

The log file for named.d also show that the queries are happening to retrieve the _acme-challenge TXT string. (I will show a representative sample of the log output from just one host -

13-Mar-2019 21:24:36.750 queries: info: client @0x7f0afc2303b0 205.151.255.18#46281 (_acme-challenge.www.domesweetdome.us.com): view localhost_resolver: query: _acme-challenge.www.domesweetdome.us.com IN SOA - (205.151.255.18)
13-Mar-2019 21:24:36.751 queries: info: client @0x7f0afc2303b0 205.151.255.18#42065 (www.domesweetdome.us.com): view localhost_resolver: query: www.domesweetdome.us.com IN SOA - (205.151.255.18)
13-Mar-2019 21:24:36.751 queries: info: client @0x7f0afc2303b0 205.151.255.18#46049 (domesweetdome.us.com): view localhost_resolver: query: domesweetdome.us.com IN SOA - (205.151.255.18)
13-Mar-2019 21:24:36.753 update: info: client @0x7f0afc2953d0 205.151.255.18#57530/key letsencrypt: view localhost_resolver: updating zone ‘domesweetdome.us.com/IN’: adding an RR at ‘_acme-challenge.www.domesweetdome.us.com’ TXT “W-qQO7fqGNKKLFc0UKLm9_jI0X2a2o-dyMKTQsfnsyo”

…
13-Mar-2019 21:24:47.002 queries: info: client @0x7f0afc2303b0 205.151.255.18#53712 (_acme-challenge.www.domesweetdome.us.com): view localhost_resolver: query: _acme-challenge.www.domesweetdome.us.com IN SOA - (205.151.255.18)
13-Mar-2019 21:24:47.002 queries: info: client @0x7f0afc2303b0 205.151.255.18#55558 (www.domesweetdome.us.com): view localhost_resolver: query: www.domesweetdome.us.com IN SOA - (205.151.255.18)
13-Mar-2019 21:24:47.003 queries: info: client @0x7f0afc2303b0 205.151.255.18#35248 (domesweetdome.us.com): view localhost_resolver: query: domesweetdome.us.com IN SOA - (205.151.255.18)
13-Mar-2019 21:24:47.004 update: info: client @0x7f0ad0015da0 205.151.255.18#57544/key letsencrypt: view localhost_resolver: updating zone ‘domesweetdome.us.com/IN’: deleting an RR at _acme-challenge.www.domesweetdome.us.com TXT

So to my untrained eyes, it seems this should be working, yet I am getting a failure and am unable to get/update any certificates. Any kind guru got a light to shed on this and help me find some joy? :wink:
Marc…

1 Like

FWIW…

$ dig +nssearch domesweetdome.us.com
SOA ns1.domesweetdome.us.com. marc.marcchamberlin.com. 2019031201 10800 3600 1209600 3600 from server 208.94.148.13 in 1 ms.
SOA ns1.domesweetdome.us.com. marc.marcchamberlin.com. 2019031201 10800 3600 1209600 3600 from server 2600:1800:5::1 in 1 ms.
SOA ns1.domesweetdome.us.com. marc.marcchamberlin.com. 2019031201 10800 3600 1209600 3600 from server 2600:1801:6::1 in 1 ms.
SOA ns1.domesweetdome.us.com. marc.marcchamberlin.com. 2019031201 10800 3600 1209600 3600 from server 208.80.124.13 in 1 ms.
SOA ns1.domesweetdome.us.com. marc.marcchamberlin.com. 2019031201 10800 3600 1209600 3600 from server 208.80.126.13 in 1 ms.
SOA ns1.domesweetdome.us.com. marc.marcchamberlin.com. 2019031201 10800 3600 1209600 3600 from server 2600:1802:7::1 in 1 ms.
;; connection timed out; no servers could be reached

I get an older serial number from DNS Made Easy, and can’t access ns1.domesweetdome.us.com. (DNSViz says it gets “connection refused”.) And ns2.marcchamberlin.com doesn’t resolve.

Maybe your primary nameserver can’t be accessed from the Internet?

And the secondary DNS Made Easy nameservers aren’t getting updated?

Then Let’s Encrypt would consistently get NXDOMAIN from the secondary nameservers.

Hi Matt, Thanks for looking into my DNS configuration for me, I think you might have caught me in the middle of doing some testing/hacking while I am trying to figure out what is going on… So I have been dinking around with some of the settings to try and observe their effect and help me to grok things better. I think I have most of the problems you were pointing out fixed but still no joy getting the challenges to working…

Give it another shot if you like… Marc…

Ping? Any ideas? Hoping my question doesn’t get lost… :wink:

I visited your site and it normally connected to your site with LE cert signed at 3/3 and I wonder why you can’t use http-01?

Any chance you can run with --debug-challenges and just cancel it instead of continuing, so it doesn’t delete the TXT records from your nameservers?

Then you can do a diagnosis of whether they actually resolve, using external tools.

Hi Orangepizza - I need a single wildcard certificate to use with my Apache James email server. Apache James, and most email servers/clients do not support SNL like most web servers do. The only way I can get a wildcard certificate is to use a DNS-01 challenge.

Hi az - If you look at my original posting, you will see that running with --debug-challenges is exactly what I did. At the point that the script pauses, I also used dig to verify if the TXT record got inserted at that point. It did… I can also see in the logs that LetsEncrypt comes back with a query later on to verify the existence of the TXT record, but for some reason I don’t understand, it is failing to issue me a certificate.

that list of domains, while it’s long list, doesn’t have wildcard domain “*. domesweetdome.us.com”. you can use http-01 if set it up correctly.

I want to check, which is why I wanted you to abort when --debug-challenges hits, instead of continuing.

Not that I don't trust you, but if you need help from others, this would be a good way to enable others to do that.

No not yet, but it will… I am just in the process of setting things up and learning the ropes of doing a DNS-01 challenge… This was a simplified example to demonstrate the problem…

then you need just two challenge one for wildcard *.domesweetdome.us.com and domesweetdome.su.com as wildcard doesn’t include base main domain. and two txt record at
_acme-challenge.domesweetdome.us.com

Hi Az, hmmm you may be on to something and I am now even more puzzled… In order to let you take a look at the records I realized I was going to have to take out the --dry-run option so as to actually get LetsEncrypt to put the TXT record in the external view. So I did that thinking I could just let it sit at the point that it pauses and then have you take a look at it for yourself. BUT I discovered that the jnl file is STILL being created in the local view, so something fishy is happening there as well… I am poking at it but may have to get back to you tomorrow as it is getting pretty late here…

Thanks again orangepizza for your suggestion, and yes I understand, and not only just for domesweetdome.us.com but for a lot of other domains that I support as well! :slight_smile: If I enumerated them all it would be a very long list indeed! Using wildcards will shorten that list dramatically! LOL

Oh crud, trying to test the external view is going to cause me to hit the rate limit…

--dry-run still applies the DNS changes as it would in live … it just uses the non-production certificate authority.

Oh OK, I will go back to using --dry-run, but then I have to figure out why the jnl file and TXT record is being created in my local view and not in my external view. I was thinking that was an effect of --dry-run, but you are implying that is not the case. So how does one control which view is being affected by the certbot renew command? Do I have to physically be doing this from a remote location and not on my own internal server/network?

That’s just a side-effect of how you have your zonefiles and keys and allow-update configured, right?

Certbot has no say in it, it just does a plain ordinary RFC2136 update. It’s up to bind to evaluate what changes are to be made.

Maybe try move the allow-update from your internal view to the external view?

Hi again Az, I turned off the permissions to allow updates in both my local and internal views, and just allow it in the external view. No joy, Certbot complained that the connection is being refused -

Saving debug log to /var/log/certbot/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for domesweetdome.us.com
dns-01 challenge for ftp.domesweetdome.us.com
dns-01 challenge for imap.domesweetdome.us.com
dns-01 challenge for mail.domesweetdome.us.com
dns-01 challenge for pop3.domesweetdome.us.com
dns-01 challenge for smtp.domesweetdome.us.com
dns-01 challenge for www.domesweetdome.us.com
Cleaning up challenges
Encountered exception during recovery
Received response from server: REFUSED
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/error_handler.py", line 103, in _call_registered
self.funcs-1
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 308, in _cleanup_challenges
self.auth.cleanup(achalls)
File "/usr/lib/python2.7/site-packages/certbot/plugins/dns_common.py", line 76, in cleanup
self._cleanup(domain, validation_domain_name, validation)
File "/usr/lib/python2.7/site-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 79, in _cleanup
self._get_rfc2136_client().del_txt_record(validation_name, validation)
File "/usr/lib/python2.7/site-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 170, in del_txt_record
.format(dns.rcode.to_text(rcode)))
PluginError: Received response from server: REFUSED
Received response from server: REFUSED

Well, okay, but if you’d like help with your BIND config, we’ll probably need to see it.