DNS problem: NXDOMAIN looking up TXT for

Hello. I'm in the process of trying to get DNS-based certificate issuing set up for a domain. I'm extending the acme4j client with support for Vultr DNS. I'm performing the DNS-01 challenge, because I may not be exposing a web server, and I want to issue at least one wildcard certificate.

My problem is that I'm creating a TXT record in response to a challenge, and:

  • This TXT record appears to be correct, and has the correct name.
  • The TXT record appears to have been propagated to various servers: After I create the record, I wait for five minutes and then make a request to 8.8.8.8 and ask it for the TXT records for the domain, and there's a record present with the correct name and digest value. I've tried waiting for longer, but this makes no difference.
  • I've also verified that the TXT record is visible from other servers, such as the authoritative nameserver for my domain (ns1.vultr.com), and various other freely accessible DNS servers.
  • I've verified that I have both A and AAAA records for the domain and the subdomain I'm trying to get a certificate for.

The challenge always fails with:

{
  "identifier": {"type":"dns","value":"www.eigion.one"},
  "status":"invalid",
  "expires":"2022-06-29T15:07:41Z",
  "challenges":[{
  "type":"dns-01",
  "status":"invalid",
  "error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.eigion.one - check that a DNS record exists for this domain","status":400},"url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/122473436266/CYSJsw","token":"xP5O17o6Yz8SnVzfmlaNdEreE4J1EZnzkwHBf5TRLIQ","validated":"2022-06-22T15:12:42Z"}]}

I've left the temporary TXT record in place after the most recent one, if anyone wants to try to inspect the DNS records.

My domain is: eigion.one

I ran this command: (Programmatically invoked acme4j library)

It produced this output:

The complete output including all of the requests made to all servers:

I've executed variations on this across multiple servers and all give the same response:

$ dig _acme_challenge.www.eigion.one txt @ns1.vultr.com
; <<>> DiG 9.16.27 <<>> _acme_challenge.www.eigion.one txt @ns1.vultr.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50650
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme_challenge.www.eigion.one.	IN	TXT

;; ANSWER SECTION:
_acme_challenge.www.eigion.one.	600 IN	TXT	"DCHAlXY2WUSog3p6beqjMQFfzjXG2JzlhnoZYZm4N1s"

;; Query time: 0 msec
;; SERVER: 2001:19f0:ccd::1#53(2001:19f0:ccd::1)
;; WHEN: Wed Jun 22 15:20:24 UTC 2022
;; MSG SIZE  rcvd: 115

My web server is (include version): Nonexistent.

The operating system my web server runs on is (include version): FreeBSD 13.

My hosting provider, if applicable, is: Vultr

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): acme4j 2.13

I should add: I posted the above command output without knowing if there are any values in there that are "private". It doesn't matter if there are, because I'll be wiping the account and domain keys entirely after I get a successful issue of a certificate.

edit: Also note that there's a 300 second pause at line 154. It sort of gets lost in all of the output, but that's where I wait for records to propagate.

Did you by any chance remove the TXT record again? Because I can't resolve it:

osiris@erazer ~ $ dig +trace _acme-challenge.www.eigion.one TXT

; <<>> DiG 9.16.27 <<>> +trace _acme-challenge.www.eigion.one TXT
;; global options: +cmd
.			85599	IN	NS	b.root-servers.net.
.			85599	IN	NS	h.root-servers.net.
.			85599	IN	NS	f.root-servers.net.
.			85599	IN	NS	e.root-servers.net.
.			85599	IN	NS	g.root-servers.net.
.			85599	IN	NS	i.root-servers.net.
.			85599	IN	NS	j.root-servers.net.
.			85599	IN	NS	l.root-servers.net.
.			85599	IN	NS	m.root-servers.net.
.			85599	IN	NS	a.root-servers.net.
.			85599	IN	NS	k.root-servers.net.
.			85599	IN	NS	d.root-servers.net.
.			85599	IN	NS	c.root-servers.net.
.			85599	IN	RRSIG	NS 8 0 518400 20220705050000 20220622040000 47671 . Vydnz2wv+NPE4OviUgFUE3BD88Q6WsSBuV/+Zg3qUFsBrGXZkMlmyeV+ +0+1dmkY7Zy7loapsHbLczPj9Q2zjJpAZMvh2hDg+ivBV6On8EYb17ej fNpdo3mrGdi5721nrslgAxErPRvJ7fJ+CaYPLdhQWDd4syyS1PsFd2St srYsOnViB3+LqftYkuANWz5yDrpjmTTC90rxop18oHV/KvLrV2a201DR uyprwolDZD/bGbl+ICf5eqq0WsCkt+EapTdSJ8EfCaUILkvz8qQwkwRq TWj9jvLSzvNslkFG0rl/J5goiZJ0ueJuJ9MjAr/gxeySM5bZjQdoSc7C ybRCPg==
;; Received 525 bytes from 2001:b88:1002::10#53(2001:b88:1002::10) in 18 ms

one.			172800	IN	NS	c.nic.one.
one.			172800	IN	NS	b.nic.one.
one.			172800	IN	NS	a.nic.one.
one.			172800	IN	NS	d.nic.one.
one.			86400	IN	DS	56899 8 2 C28CDE2F5E88289C26584A4E6871B41E0F0C2D5F7F2DF2906E8FBB27 5AC4AA68
one.			86400	IN	DS	14131 8 1 8C04B443EE763B8B67CDF0DB0BBC832E24F560EE
one.			86400	IN	DS	14131 8 2 8D11FF81A0E9BCC2719695CBE4D585B47AA3BDE6CD28C5AC6E02BD91 9CA9B9E0
one.			86400	IN	RRSIG	DS 8 1 86400 20220705050000 20220622040000 47671 . baJtpeGttU7DKrhg+diiERZI/7aXTN+ub1qQOlsrix7I9xEqZ82GuM+G TDyma6Qakov15bfe+SKPqEPsCB48kV61GA5lYGe53ksmeU51TjvwzRcn TLi8M8EhXsjV+CG7TjPOAU6H4hWNUCs3+RTqqb6xPfiWbEt+V/EZLzym ppMGnI9UrNdegOOhN8gWXx+Go5bo1uwxR3/vF8ALS1reYmGP2E21wr3s u9iRce5dqNBk4+3qzusgcS0mj+ffhkPJrB/DZolLTgtUkVpTfBNj8j/4 3SqWDtZ7lrHGR8cLlFrukCzgg/bIfTt76jr9jhZlRd1ztIY9FablWFvo kJnyOg==
;; Received 722 bytes from 198.41.0.4#53(a.root-servers.net) in 16 ms

eigion.one.		3600	IN	NS	ns2.vultr.com.
eigion.one.		3600	IN	NS	ns1.vultr.com.
f4cmn012br11ovo9u99g0pmvcn9bqfs7.one. 1800 IN NSEC3 1 1 1 4C7D8E66 F4DB4E286DME626SJD58CV1CCANJN743 NS SOA RRSIG DNSKEY NSEC3PARAM
f4cmn012br11ovo9u99g0pmvcn9bqfs7.one. 1800 IN RRSIG NSEC3 8 2 1800 20220720093448 20220620093018 4353 one. eX8xAd3hFXu1AyJToi8Ee23reScog1qDfa48xAh/MUWbeRWxPfee53su y9WwWeQGZu23tZKbQcmoPoApzaGp1vEcyP0XFJi424Q6UeL/JEjPqhP8 IrHd1H7k/7QrrG0sx2N5FD4ric2Wb/chQjzD/pEpH130U8OPPw5qOOrn OcGfWIMZlSfWMgUl+gTsSCSUKI03BD6iWOgSJ3RuRs0Ftw==
s7sf0sm463qjnlc9pprsj38jr1j797i1.one. 1800 IN NSEC3 1 1 1 4C7D8E66 S7T0356VGLTNNHF09KQL4K9J9KNS05LC NS DS RRSIG
s7sf0sm463qjnlc9pprsj38jr1j797i1.one. 1800 IN RRSIG NSEC3 8 2 1800 20220716173607 20220616173049 4353 one. pIFWNpLkJ9CJ8Q6x0U4r0OS1BcLcTvT7V8xUfb7J3p4EnKhk7XpFraTT IvJcxAVx/Pj1j99UBC+MPIeiUEY9Oig3ssvfJQajzpTePn+pf1BeluCU 5fqZvrzOriY196fKuE1jM5EB8c8MKRUy8WF/RBMLo0dv9CQwmvgSyJQ5 iSiI5wCkdy9OnCtlz9dSQ/VcxCC1zkK0E1y1ZexD4NDdqg==
;; Received 671 bytes from 2001:dcd:3::9#53(c.nic.one) in 18 ms

eigion.one.		300	IN	SOA	ns1.vultr.com. dnsadm.choopa.com. 0 10800 3600 604800 3600
;; Received 122 bytes from 173.199.96.96#53(ns1.vultr.com) in 18 ms

osiris@erazer ~ $ 

There shouldn't be anything private in those logs. All the key-related stuff should be public keys and/or signatures.

Aahh, I see it! You're using an underscore where it should be a dash. See the difference:

  • _acme_challenge
  • _acme-challenge

:slight_smile:

5 Likes

Oh ... dear ...

Let's see what happens after I change that...

THANK YOU!

That did it. I can't believe that's what the problem was. :weary:

3 Likes

Why do you need to input the label manually anyway? Isn't _acme-challenge hardcoded in acmej4?

2 Likes

acme4j just returns the domain and the required digest value. It assumes that you'll either create the DNS record manually via a control panel, or call a provider's DNS API (like I'm doing). The record name doesn't seem to appear in the code anywhere.

1 Like

Hmkay, sounds like a terrible implementation. When presenting the token value, why not also present the entire FQDN required, including the _acme-challenge label? Weird..

Ah well, all sorted out again now.

1 Like

acme4j is taking care of the ACME protocol, so it's not a full-featured client that also takes care of setting DNS records, or configuring a HTTP server to deliver the correct challenge response. This design choice is fully intentional. I didn't want to design a client that expects certain services to be present, or modifies server configuration files. Instead it's left to the user of acme4j to set the DNS server correctly. I assume that the user knows better how to change configuration of their infrastructure.

Giving the entire FQDN with the Dns01Challenge is a great idea. The problem is that the ACME protocol does not send the affected domain along with the challenge. So from the perspective of a challenge, it's not possible to know what domain is actually being challenged by it.

This is all documented in details in the acme4j docs, and even in the example source code. I don't think it's fair to call it a "terrible implementation".

1 Like

Aha. I guess I was thrown off by the following sentence in the README (bold emphasize by me):

This Java client helps connecting to an ACME server, and performing all necessary steps to manage certificates.

To me, this suggests acme4j can do everything to also issue a certificate, which includes doing challenges.

Correct, the challenge itself doesn't have that info, but a challenge is always part of a single authorization which does have the identifier information. Not that difficult to combine both in memory of the application I recon. Doesn't the application combine both infos already? How would it know which challenge to perform?

For example, the acme library which is the basis for the Certbot client has it hardcoded in the challenge type:

So not only the user doesn't have to worry about the prepended label, but also any client making use of the acme library doesn't have to worry about it.

Might have been a little bit harsh. That said, OP said the client provided the domain and token, but OP had to manually add _acme-challenge. IMO that detail is not a good implementation to be honest. Terrible might be a little bit strong though, I'm sure you're doing your best :slight_smile: Also looking at the amount of stars on the Github repo, I'm also probably the only one calling it terrible :wink:

By the way, I'm a little bit puzzled. The acme4j client is listed under "Libraries" on the ACME Client Implementations - Let's Encrypt page and not "Clients"? But the repo README clearly states its a client? Or should I see it as a client/server idea with acme4j being a library offering an ACME client?

2 Likes