DNS problem: NXDOMAIN looking up TXT for

io7m · June 22, 2022, 3:40pm

Hello. I'm in the process of trying to get DNS-based certificate issuing set up for a domain. I'm extending the acme4j client with support for Vultr DNS. I'm performing the DNS-01 challenge, because I may not be exposing a web server, and I want to issue at least one wildcard certificate.

My problem is that I'm creating a TXT record in response to a challenge, and:

This TXT record appears to be correct, and has the correct name.
The TXT record appears to have been propagated to various servers: After I create the record, I wait for five minutes and then make a request to 8.8.8.8 and ask it for the TXT records for the domain, and there's a record present with the correct name and digest value. I've tried waiting for longer, but this makes no difference.
I've also verified that the TXT record is visible from other servers, such as the authoritative nameserver for my domain (ns1.vultr.com), and various other freely accessible DNS servers.
I've verified that I have both A and AAAA records for the domain and the subdomain I'm trying to get a certificate for.

The challenge always fails with:

{
  "identifier": {"type":"dns","value":"www.eigion.one"},
  "status":"invalid",
  "expires":"2022-06-29T15:07:41Z",
  "challenges":[{
  "type":"dns-01",
  "status":"invalid",
  "error":{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.eigion.one - check that a DNS record exists for this domain","status":400},"url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/122473436266/CYSJsw","token":"xP5O17o6Yz8SnVzfmlaNdEreE4J1EZnzkwHBf5TRLIQ","validated":"2022-06-22T15:12:42Z"}]}

I've left the temporary TXT record in place after the most recent one, if anyone wants to try to inspect the DNS records.

My domain is: eigion.one

I ran this command: (Programmatically invoked acme4j library)

It produced this output:

The complete output including all of the requests made to all servers:

gist.github.com

https://gist.github.com/io7m/295bcd8692dd5a47478789152c66e640

gistfile1.txt

DEBUG com.io7m.certusine.vanilla.internal.CSDomainExecutor executing renewal for domain eigion.one
DEBUG com.io7m.certusine.vanilla.internal.CSDomainExecutor locating account
DEBUG org.shredzone.acme4j.AccountBuilder create
DEBUG org.shredzone.acme4j.connector.DefaultConnection GET https://acme-v02.api.letsencrypt.org/directory
DEBUG org.shredzone.acme4j.connector.DefaultConnection HEADER X-Frame-Options: DENY
DEBUG org.shredzone.acme4j.connector.DefaultConnection HEADER null: HTTP/1.1 200 OK
DEBUG org.shredzone.acme4j.connector.DefaultConnection HEADER Strict-Transport-Security: max-age=604800
DEBUG org.shredzone.acme4j.connector.DefaultConnection HEADER Cache-Control: public, max-age=0, no-cache
DEBUG org.shredzone.acme4j.connector.DefaultConnection HEADER Server: nginx
DEBUG org.shredzone.acme4j.connector.DefaultConnection HEADER Connection: keep-alive

This file has been truncated. show original

I've executed variations on this across multiple servers and all give the same response:

$ dig _acme_challenge.www.eigion.one txt @ns1.vultr.com
; <<>> DiG 9.16.27 <<>> _acme_challenge.www.eigion.one txt @ns1.vultr.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50650
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme_challenge.www.eigion.one.	IN	TXT

;; ANSWER SECTION:
_acme_challenge.www.eigion.one.	600 IN	TXT	"DCHAlXY2WUSog3p6beqjMQFfzjXG2JzlhnoZYZm4N1s"

;; Query time: 0 msec
;; SERVER: 2001:19f0:ccd::1#53(2001:19f0:ccd::1)
;; WHEN: Wed Jun 22 15:20:24 UTC 2022
;; MSG SIZE  rcvd: 115

My web server is (include version): Nonexistent.

The operating system my web server runs on is (include version): FreeBSD 13.

My hosting provider, if applicable, is: Vultr

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): acme4j 2.13

io7m · June 22, 2022, 3:41pm

I should add: I posted the above command output without knowing if there are any values in there that are "private". It doesn't matter if there are, because I'll be wiping the account and domain keys entirely after I get a successful issue of a certificate.

edit: Also note that there's a 300 second pause at line 154. It sort of gets lost in all of the output, but that's where I wait for records to propagate.

Osiris · June 22, 2022, 3:53pm

Did you by any chance remove the TXT record again? Because I can't resolve it:

osiris@erazer ~ $ dig +trace _acme-challenge.www.eigion.one TXT

; <<>> DiG 9.16.27 <<>> +trace _acme-challenge.www.eigion.one TXT
;; global options: +cmd
.			85599	IN	NS	b.root-servers.net.
.			85599	IN	NS	h.root-servers.net.
.			85599	IN	NS	f.root-servers.net.
.			85599	IN	NS	e.root-servers.net.
.			85599	IN	NS	g.root-servers.net.
.			85599	IN	NS	i.root-servers.net.
.			85599	IN	NS	j.root-servers.net.
.			85599	IN	NS	l.root-servers.net.
.			85599	IN	NS	m.root-servers.net.
.			85599	IN	NS	a.root-servers.net.
.			85599	IN	NS	k.root-servers.net.
.			85599	IN	NS	d.root-servers.net.
.			85599	IN	NS	c.root-servers.net.
.			85599	IN	RRSIG	NS 8 0 518400 20220705050000 20220622040000 47671 . Vydnz2wv+NPE4OviUgFUE3BD88Q6WsSBuV/+Zg3qUFsBrGXZkMlmyeV+ +0+1dmkY7Zy7loapsHbLczPj9Q2zjJpAZMvh2hDg+ivBV6On8EYb17ej fNpdo3mrGdi5721nrslgAxErPRvJ7fJ+CaYPLdhQWDd4syyS1PsFd2St srYsOnViB3+LqftYkuANWz5yDrpjmTTC90rxop18oHV/KvLrV2a201DR uyprwolDZD/bGbl+ICf5eqq0WsCkt+EapTdSJ8EfCaUILkvz8qQwkwRq TWj9jvLSzvNslkFG0rl/J5goiZJ0ueJuJ9MjAr/gxeySM5bZjQdoSc7C ybRCPg==
;; Received 525 bytes from 2001:b88:1002::10#53(2001:b88:1002::10) in 18 ms

one.			172800	IN	NS	c.nic.one.
one.			172800	IN	NS	b.nic.one.
one.			172800	IN	NS	a.nic.one.
one.			172800	IN	NS	d.nic.one.
one.			86400	IN	DS	56899 8 2 C28CDE2F5E88289C26584A4E6871B41E0F0C2D5F7F2DF2906E8FBB27 5AC4AA68
one.			86400	IN	DS	14131 8 1 8C04B443EE763B8B67CDF0DB0BBC832E24F560EE
one.			86400	IN	DS	14131 8 2 8D11FF81A0E9BCC2719695CBE4D585B47AA3BDE6CD28C5AC6E02BD91 9CA9B9E0
one.			86400	IN	RRSIG	DS 8 1 86400 20220705050000 20220622040000 47671 . baJtpeGttU7DKrhg+diiERZI/7aXTN+ub1qQOlsrix7I9xEqZ82GuM+G TDyma6Qakov15bfe+SKPqEPsCB48kV61GA5lYGe53ksmeU51TjvwzRcn TLi8M8EhXsjV+CG7TjPOAU6H4hWNUCs3+RTqqb6xPfiWbEt+V/EZLzym ppMGnI9UrNdegOOhN8gWXx+Go5bo1uwxR3/vF8ALS1reYmGP2E21wr3s u9iRce5dqNBk4+3qzusgcS0mj+ffhkPJrB/DZolLTgtUkVpTfBNj8j/4 3SqWDtZ7lrHGR8cLlFrukCzgg/bIfTt76jr9jhZlRd1ztIY9FablWFvo kJnyOg==
;; Received 722 bytes from 198.41.0.4#53(a.root-servers.net) in 16 ms

eigion.one.		3600	IN	NS	ns2.vultr.com.
eigion.one.		3600	IN	NS	ns1.vultr.com.
f4cmn012br11ovo9u99g0pmvcn9bqfs7.one. 1800 IN NSEC3 1 1 1 4C7D8E66 F4DB4E286DME626SJD58CV1CCANJN743 NS SOA RRSIG DNSKEY NSEC3PARAM
f4cmn012br11ovo9u99g0pmvcn9bqfs7.one. 1800 IN RRSIG NSEC3 8 2 1800 20220720093448 20220620093018 4353 one. eX8xAd3hFXu1AyJToi8Ee23reScog1qDfa48xAh/MUWbeRWxPfee53su y9WwWeQGZu23tZKbQcmoPoApzaGp1vEcyP0XFJi424Q6UeL/JEjPqhP8 IrHd1H7k/7QrrG0sx2N5FD4ric2Wb/chQjzD/pEpH130U8OPPw5qOOrn OcGfWIMZlSfWMgUl+gTsSCSUKI03BD6iWOgSJ3RuRs0Ftw==
s7sf0sm463qjnlc9pprsj38jr1j797i1.one. 1800 IN NSEC3 1 1 1 4C7D8E66 S7T0356VGLTNNHF09KQL4K9J9KNS05LC NS DS RRSIG
s7sf0sm463qjnlc9pprsj38jr1j797i1.one. 1800 IN RRSIG NSEC3 8 2 1800 20220716173607 20220616173049 4353 one. pIFWNpLkJ9CJ8Q6x0U4r0OS1BcLcTvT7V8xUfb7J3p4EnKhk7XpFraTT IvJcxAVx/Pj1j99UBC+MPIeiUEY9Oig3ssvfJQajzpTePn+pf1BeluCU 5fqZvrzOriY196fKuE1jM5EB8c8MKRUy8WF/RBMLo0dv9CQwmvgSyJQ5 iSiI5wCkdy9OnCtlz9dSQ/VcxCC1zkK0E1y1ZexD4NDdqg==
;; Received 671 bytes from 2001:dcd:3::9#53(c.nic.one) in 18 ms

eigion.one.		300	IN	SOA	ns1.vultr.com. dnsadm.choopa.com. 0 10800 3600 604800 3600
;; Received 122 bytes from 173.199.96.96#53(ns1.vultr.com) in 18 ms

osiris@erazer ~ $

There shouldn't be anything private in those logs. All the key-related stuff should be public keys and/or signatures.

Aahh, I see it! You're using an underscore where it should be a dash. See the difference:

_acme_challenge
_acme-challenge

io7m · June 22, 2022, 4:09pm

Oh ... dear ...

Let's see what happens after I change that...

io7m · June 22, 2022, 4:16pm

THANK YOU!

That did it. I can't believe that's what the problem was.

Osiris · June 22, 2022, 4:17pm

Why do you need to input the label manually anyway? Isn't _acme-challenge hardcoded in acmej4?

io7m · June 22, 2022, 4:24pm

acme4j just returns the domain and the required digest value. It assumes that you'll either create the DNS record manually via a control panel, or call a provider's DNS API (like I'm doing). The record name doesn't seem to appear in the code anywhere.

Osiris · June 22, 2022, 4:53pm

Hmkay, sounds like a terrible implementation. When presenting the token value, why not also present the entire FQDN required, including the _acme-challenge label? Weird..

Ah well, all sorted out again now.

shred · June 28, 2022, 11:56am

acme4j is taking care of the ACME protocol, so it's not a full-featured client that also takes care of setting DNS records, or configuring a HTTP server to deliver the correct challenge response. This design choice is fully intentional. I didn't want to design a client that expects certain services to be present, or modifies server configuration files. Instead it's left to the user of acme4j to set the DNS server correctly. I assume that the user knows better how to change configuration of their infrastructure.

Giving the entire FQDN with the Dns01Challenge is a great idea. The problem is that the ACME protocol does not send the affected domain along with the challenge. So from the perspective of a challenge, it's not possible to know what domain is actually being challenged by it.

This is all documented in details in the acme4j docs, and even in the example source code. I don't think it's fair to call it a "terrible implementation".

Osiris · June 28, 2022, 12:06pm

Aha. I guess I was thrown off by the following sentence in the README (bold emphasize by me):

This Java client helps connecting to an ACME server, and performing all necessary steps to manage certificates.

To me, this suggests acme4j can do everything to also issue a certificate, which includes doing challenges.

Correct, the challenge itself doesn't have that info, but a challenge is always part of a single authorization which does have the identifier information. Not that difficult to combine both in memory of the application I recon. Doesn't the application combine both infos already? How would it know which challenge to perform?

For example, the acme library which is the basis for the Certbot client has it hardcoded in the challenge type:

github.com

certbot/certbot/blob/b9f9952660ac63acf7ea5abe729f0d378362e5b4/acme/acme/challenges.py#L262


      
                      logger.debug("Verification of key authorization in response failed")
                  return verified
          
          
          @Challenge.register
          class DNS01(KeyAuthorizationChallenge):
              """ACME dns-01 challenge."""
              response_cls = DNS01Response
              typ = response_cls.typ
          
              LABEL = "_acme-challenge"
              """Label clients prepend to the domain name being validated."""
          
              def validation(self, account_key: jose.JWK, **unused_kwargs: Any) -> str:
                  """Generate validation.
          
                  :param JWK account_key:
                  :rtype: str
          
                  """
                  return jose.b64encode(hashlib.sha256(self.key_authorization(

So not only the user doesn't have to worry about the prepended label, but also any client making use of the acme library doesn't have to worry about it.

Might have been a little bit harsh. That said, OP said the client provided the domain and token, but OP had to manually add _acme-challenge. IMO that detail is not a good implementation to be honest. Terrible might be a little bit strong though, I'm sure you're doing your best Also looking at the amount of stars on the Github repo, I'm also probably the only one calling it terrible

By the way, I'm a little bit puzzled. The acme4j client is listed under "Libraries" on the ACME Client Implementations - Let's Encrypt page and not "Clients"? But the repo README clearly states its a client? Or should I see it as a client/server idea with acme4j being a library offering an ACME client?

system · July 28, 2022, 12:07pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
[Solved]DNS problem: NXDOMAIN looking up TXT for _acme-challenge Help	5	79963	September 21, 2018
NXDOMAIN looking up TXT for _acme-challenge.mydomain.net Help	6	3589	April 2, 2020
DNS challenge failing: NXDOMAIN looking up TXT Issuance Tech	25	41620	August 15, 2016
DNS problem: NXDOMAIN looking up TXT for _acme-challenge Help	3	5263	February 26, 2020
DNS problem: NXDOMAIN looking up TXT error for a wildcard cert request Help	9	1914	January 26, 2021

DNS problem: NXDOMAIN looking up TXT for

Related topics