Incorrect text validation

Running on opnsense using DNS 01 with Powerdns. Acme generates the text record, successfully passes it to PDNS, which then zone transfers to HE.net.

With a 5-10 minute wait time (I've tested), acme always comes back with an incorrect record

I had valid certs up until my pfsense box died on me, and I didn't have backups of the old certs

My domain is: wapnitsky.com

I ran this command: acme cert reissue

It produced this output:

See attached file

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

acmeclient_20220224.log.txt (49.3 KB)

No record should be expected, but incorrect?

Do you recognize the record? Is it an old one? Old records should be removed by the script, once they've exhausted their usefulness.

Also, when you get a certificate for example.com,*.example.com you put two records on _acme-challenge.example.com and the validator still works. It doesn't matter how many records there are, as long as the right one is there.

1 Like

You have many nameservers for your domain and they all need to have the same TXT record present for validation to definitely pass (you can get lucky if just the right ones are updated):

wapnitsky.com.          0       IN      NS      ns4.he.net.
wapnitsky.com.          0       IN      NS      Ns4.afraid.org.
wapnitsky.com.          0       IN      NS      ns2.he.net.
wapnitsky.com.          0       IN      NS      ns3.afraid.org.
wapnitsky.com.          0       IN      NS      ns2.afraid.org.
wapnitsky.com.          0       IN      NS      ns5.he.net.
wapnitsky.com.          0       IN      NS      ns-global.kjsl.com.
wapnitsky.com.          0       IN      NS      uz5qfm8n244kn4qz8mh437w9kzvpudduwyldp5361v9n0vh8sx5ucu.free.ns.buddyns.com.
wapnitsky.com.          0       IN      NS      uz5x36jqv06q5yulzwcblfzcrk1b479xdttdm1nrgfglzs57bmctl8.free.ns.buddyns.com.
wapnitsky.com.          0       IN      NS      ns1.he.net.
wapnitsky.com.          0       IN      NS      Ns1.afraid.org.
wapnitsky.com.          0       IN      NS      ns3.he.net.
wapnitsky.com.          0       IN      NS      uz588h0rhwuu3cc03gm9uckw0w42cqr459wn1nxrbzhym2wd81zydb.free.ns.buddyns.com.
3 Likes

Out of interest, why do you have so many?

3 Likes

To me, it also looks like there are multiple DNS hosters present?

2 Likes

If they're all synced it shouldn't be a problem. But some aren't responding at all.

~ $ for ns in $(dig +short ns wapnitsky.com); do dig +short @$ns txt _acme-challenge.wapnitsky.com ; done
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
;; connection timed out; no servers could be reached

"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"
~ $ dig +short ns wapnitsky.com | wc -l                    
13
2 Likes

Backup DNS, but I can remove some. Was testing in my home lab

Removed extra servers, but received the same issue on a different domain with only thr he.net servers - qual-itsystems.com

Some of your servers aren't responding and some are responding with no record.

I still want to understand if the record that appears is something that makes sense or not. Is it right, but not everywhere, or it's just wrong?

~$ for ns in $(dig +short ns wapnitsky.com); do echo "$ns: " ; dig +short @$ns
txt _acme-challenge.wapnitsky.com ; echo ; done
ns3.afraid.org.:

ns4.he.net.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

uz5x36jqv06q5yulzwcblfzcrk1b479xdttdm1nrgfglzs57bmctl8.free.ns.buddyns.com.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

uz588h0rhwuu3cc03gm9uckw0w42cqr459wn1nxrbzhym2wd81zydb.free.ns.buddyns.com.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

uz5qfm8n244kn4qz8mh437w9kzvpudduwyldp5361v9n0vh8sx5ucu.free.ns.buddyns.com.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

ns3.he.net.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

ns1.he.net.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

Ns1.afraid.org.:
;; connection timed out; no servers could be reached


ns5.he.net.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

Ns4.afraid.org.:

ns2.afraid.org.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

ns2.he.net.:
"6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY"

ns-global.kjsl.com.:

This looks like it should work.

~$ for ns in $(dig +short ns qual-itsystems.com); do echo "$ns: " ; dig +short
@$ns txt _acme-challenge.qual-itsystems.com ; echo ; done
ns2.he.net.:
"rDnEtoACu60GWA6h671ht0odmmlBmC4TMtYToL8GIFA"

ns3.he.net.:
"rDnEtoACu60GWA6h671ht0odmmlBmC4TMtYToL8GIFA"

ns5.he.net.:
"rDnEtoACu60GWA6h671ht0odmmlBmC4TMtYToL8GIFA"

ns1.he.net.:
"rDnEtoACu60GWA6h671ht0odmmlBmC4TMtYToL8GIFA"

ns4.he.net.:
"rDnEtoACu60GWA6h671ht0odmmlBmC4TMtYToL8GIFA"

Now, if your client is sending garbage or your chain is somewhat mangling it it's another issue. What was the record supposed to be?

1 Like

Still failing.

The record is correct. I've verified in PDNS and DNS.he.net

I read your log carefully an I found the problem. It's a bug in dns_pdns.sh that removes the record needed to validate wapnitsky.com when it adds the one to validate *.wapnitsky.com. It needs to add them both at once.

See the explanation for what "changetype": "REPLACE" does: Zones — PowerDNS Authoritative Server documentation

I assume it happened because you're not running the same versions now.

<15>1 2022-02-24T18:35:48-05:00 OPNsense.wapnet.local.lan acme.sh 12165 - [meta sequenceId="110"] [Thu Feb 24 18:35:48 EST 2022] data='{"rrsets": [{"changetype": "REPLACE", "name": "_acme-challenge.wapnitsky.com.", "type": "TXT", "ttl": 60, "records": [{"content": "\"PW2r_r87-hgnJ9CqhaZ7mZbSsuNZ6L8JpKLDgE9yxws\"", "disabled": false}]}]}'
<15>1 2022-02-24T18:35:55-05:00 OPNsense.wapnet.local.lan acme.sh 13646 - [meta sequenceId="141"] [Thu Feb 24 18:35:55 EST 2022] data='{"rrsets": [{"changetype": "REPLACE", "name": "_acme-challenge.wapnitsky.com.", "type": "TXT", "ttl": 60, "records": [{"content": "\"bXBvJV0wJIjdTy_r9SBI23LYkjpgkrAY1hozBFzYY_c\"", "disabled": false}, {"content": "\"PW2r_r87-hgnJ9CqhaZ7mZbSsuNZ6L8JpKLDgE9yxws\"", "disabled": false}]}]}'

And issue opened: dns_pdns nukes ALL txt records when adding one. · Issue #3956 · acmesh-official/acme.sh · GitHub

I didn't read carefully enough, it looks like.

2 Likes

Same versions of what? :slight_smile:

And I just migrated from pfsense to opnsense, and was doing manual cert verification prior, so I never encountered this before.

Ok, but there's something I don't understand.

There's code in there to do exacly what I expect it to do, avoid nuking all other TXT records. Why it doesn't work, I have no idea. A race condition on powerdns that doesn't make the first record available for GET before sending the second one? An actual bug in the plugin?

Things is, it's your DNS is update is broken. Let's Encrypt is working fine, it's your own process that not working, fix that and it will all work.

3 Likes

And, wait.

This call is actually correct, it's adding two records.

Why aren't there two records in your DNS? @lgwapnitsky?

1 Like

I don't know. I'm.running it via opnsense acme.

I even just tried with a single domain and it failed

I always see the same "6emS6pYwizxSASjNK2r84Df_HjRXNOVzHpped1-_DuY" record, though.

Is it possible your other nameservers aren't properly synced with powerdns?

1 Like

I created the single record, populated it to he.net, then failed on.verification. the text records were identical

You need two records, one for wildcard and one for apex.

And you should automate everything possible.

1 Like