Certificate verification failure (Certbot doesn't reach our DNS server txt records)

Dear community, we're kinescope.io - a modern video platform for business and we're happy to use LE in our CDN environment. We'll appreciate your help with the issue described below and ready to donate in case of success :heart:

Problem description

  1. We run our own DNS servers (it's an important part of our CDN routing setup) using CoreDNS software v. 1.8.3.
    ns2.kinedns.net
    ns1.kinedns.net

We're adding these txt records in order to verify the wildcard certificate:

dig _acme-challenge.kinescopecdn.net TXT
; <<>> DiG 9.10.6 <<>> _acme-challenge.kinescopecdn.net TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16524
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.kinescopecdn.net. IN TXT

;; ANSWER SECTION:
_acme-challenge.kinescopecdn.net. 3600 IN TXT "uoLgpDT_PYVfFuncVQPYhA_zCNJ4OYtM58NEnWxZZoI.IpX8HpoHwIQPuwHNGw2YSuYfJEcd9213a3MuSdzmk8Y"

_acme-challenge.kinescopecdn.net. 3600 IN TXT "Sr9VfLf6ONUWtbxqVngyxeJvYbmxDeMExj13OS6KTu8"

;; Query time: 107 msec
;; SERVER: 10.118.0.1#53(10.118.0.1)
;; WHEN: Thu May 06 11:55:06 MSK 2021
; MSG SIZE rcvd: 217

Problem: But Certbot refuses to see them

Domain: kinescopecdn.net
Type: unauthorized
Detail: No TXT record found at _acme-challenge.kinescopecdn.net

Domain: kinescopecdn.net
Type: unauthorized
Detail: No TXT record found at _acme-challenge.kinescopecdn.net

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

Our CDN domain is: *.kinescopecdn.net

We run this command:
certbot certonly -m our@email.com --agree-tos --non-interactive --manual-public-ip-logging-ok --manual --preferred-challenges=dns -d kinescopecdn.net -d *.kinescopecdn.net

By this we're adding two txt records that are visible if we check them with dig txt:

_acme-challenge.kinescopecdn.net. 116 IN TXT    "5iB9reSg5kSLn7b9ZMFZ62RSSo3Fa4H0UJvWWUeHFLk"
_acme-challenge.kinescopecdn.net. 116 IN TXT    "_QmMxyEU0Har_KcuGfdrY8obTTDrx2qGmThxdgb7YXw"

Certbot produces this output:

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: kinescopecdn.net
   Type:   unauthorized
   Detail: No TXT record found at _acme-challenge.kinescopecdn.net

   Domain: kinescopecdn.net
   Type:   unauthorized
   Detail: No TXT record found at _acme-challenge.kinescopecdn.net

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Our web server is (include version): it's not relevant in this case but we use hitch 1.7.0

The operating system my web server runs on is (include version): Ubuntu 20.04

My hosting provider, if applicable, is: Kinescoe is hosting provider (PeeringDB)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot -- 0.40.0

Hi @Kinescope

I can't see your TXT entries. Not with nslookup, not with Unboundtest ( https://unboundtest.com/m/TXT/_acme-challenge.kinescopecdn.net/NQBED5YX ).

Same your check created yesterday - https://check-your-website.server-daten.de/?q=kinescopecdn.net#txt

Looks like your internal DNS doesn't export thes informations to your external DNS.

ns1.kinedns.net
	193.238.46.22

is one external name server.

:~$ dig TXT _acme-challenge.kinescopecdn.net. @193.238.46.22

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> TXT _acme-challenge.kinescopecdn.net. @193.238.46.22
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25308
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 540a6980448e31a3 (echoed)
;; QUESTION SECTION:
;_acme-challenge.kinescopecdn.net. IN TXT

;; Query time: 58 msec
;; SERVER: 193.238.46.22#53(193.238.46.22)
;; WHEN: Thu May 06 12:09:17 CEST 2021
;; MSG SIZE rcvd: 73

No TXT visible.

1 Like

Dear @JuergenAuer

First of all thanks for your reply and interest to help.
The reason why you cannot see the TXT records is that they have a default lifetime of 120 seconds so we dig right after adding them. From now for testing purposes, we've changed this value to 7 days so you can check it now once again.

dig TXT _acme-challenge.kinescopecdn.net. @193.238.46.22

;; ANSWER SECTION:
_acme-challenge.kinescopecdn.net. 120 IN TXT    "LgvCztESa5kFHVhTnbZS9kfdLV8PG38qwrO75y3_kmk"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "qlqEKLy7d0RK878Uoz-v9CBKfnIyP38gYZtMgXTk9Rk"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "AS1fCmHObr89NtH3cmGqhZTsaJ-bKrDLrAWA7Z7-mIw"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "dOck83hNN2yn7v8hL7vpiAhhhWlGGApd8fOLYF4aE4M"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "lVrJtc3XzIeDvzlUDcI7qYHrsO4vkyW3VHiD5KvOWh8"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "V4YXbCp8DNMmY2M09rjFh83LLA9hbodEqnDov-q02xU"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "PcmroattI1YSlGavIYUGMbJiKGUaQIo0oFoZSOqYCgs"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "WGof3VEFxurzRghbZ_XFea74Yn2weJnucM8BFeDUEWA"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "lu_xajMSyADQH6wYemAD6S4YQFCREjSuNd_sb5cOclY"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "D4mcx6tDkz3d6II6xydwM4xmdxqrl95Uhdb3WFElt_w"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "l17qFMQP3l77aojnlyWeIevaLik5BOjyO0gGxWFimj4"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "Sr9VfLf6ONUWtbxqVngyxeJvYbmxDeMExj13OS6KTu8"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "uoLgpDT_PYVfFuncVQPYhA_zCNJ4OYtM58NEnWxZZoI.IpX8HpoHwIQPuwHNGw2YSuYfJEcd9213a3MuSdzmk8Y"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "5iB9reSg5kSLn7b9ZMFZ62RSSo3Fa4H0UJvWWUeHFLk"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "_QmMxyEU0Har_KcuGfdrY8obTTDrx2qGmThxdgb7YXw"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "B6tHHS4UN2ix_OvlWXKFKHZXHOZM9K_SQoACTZRMeVU"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "yyYv3sCWKugtHGnprSeI_rZhW4_Z8HV4G8QWkv0F0iA"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "5T069_eJVXZy_LNZTx4B9imWZgGM-a8cedoMfSTExQk"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "H4KW2vE-U8YPQdrUz_LmQxr4l3cRcf4pgWg6HnTkYwc"

Now:

:~$ dig TXT _acme-challenge.kinescopecdn.net. @193.238.46.22

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> TXT _acme-challenge.kinescopecdn.net. @193.238.46.22
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59814
;; flags: qr aa rd; QUERY: 1, ANSWER: 19, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e76e169ffe17f076 (echoed)
;; QUESTION SECTION:
;_acme-challenge.kinescopecdn.net. IN TXT

;; ANSWER SECTION:
_acme-challenge.kinescopecdn.net. 120 IN TXT "LgvCztESa5kFHVhTnbZS9kfdLV8PG38qwrO75y3_kmk"
_acme-challenge.kinescopecdn.net. 120 IN TXT "qlqEKLy7d0RK878Uoz-v9CBKfnIyP38gYZtMgXTk9Rk"
_acme-challenge.kinescopecdn.net. 120 IN TXT "AS1fCmHObr89NtH3cmGqhZTsaJ-bKrDLrAWA7Z7-mIw"
_acme-challenge.kinescopecdn.net. 120 IN TXT "dOck83hNN2yn7v8hL7vpiAhhhWlGGApd8fOLYF4aE4M"
_acme-challenge.kinescopecdn.net. 120 IN TXT "lVrJtc3XzIeDvzlUDcI7qYHrsO4vkyW3VHiD5KvOWh8"
_acme-challenge.kinescopecdn.net. 120 IN TXT "V4YXbCp8DNMmY2M09rjFh83LLA9hbodEqnDov-q02xU"
_acme-challenge.kinescopecdn.net. 120 IN TXT "PcmroattI1YSlGavIYUGMbJiKGUaQIo0oFoZSOqYCgs"
_acme-challenge.kinescopecdn.net. 120 IN TXT "WGof3VEFxurzRghbZ_XFea74Yn2weJnucM8BFeDUEWA"
_acme-challenge.kinescopecdn.net. 120 IN TXT "lu_xajMSyADQH6wYemAD6S4YQFCREjSuNd_sb5cOclY"
_acme-challenge.kinescopecdn.net. 120 IN TXT "D4mcx6tDkz3d6II6xydwM4xmdxqrl95Uhdb3WFElt_w"
_acme-challenge.kinescopecdn.net. 120 IN TXT "l17qFMQP3l77aojnlyWeIevaLik5BOjyO0gGxWFimj4"
_acme-challenge.kinescopecdn.net. 120 IN TXT "Sr9VfLf6ONUWtbxqVngyxeJvYbmxDeMExj13OS6KTu8"
_acme-challenge.kinescopecdn.net. 120 IN TXT "uoLgpDT_PYVfFuncVQPYhA_zCNJ4OYtM58NEnWxZZoI.IpX8HpoHwIQPuwHNGw2YSuYfJEcd9213a3MuSdzmk8Y"
_acme-challenge.kinescopecdn.net. 120 IN TXT "5iB9reSg5kSLn7b9ZMFZ62RSSo3Fa4H0UJvWWUeHFLk"
_acme-challenge.kinescopecdn.net. 120 IN TXT "_QmMxyEU0Har_KcuGfdrY8obTTDrx2qGmThxdgb7YXw"
_acme-challenge.kinescopecdn.net. 120 IN TXT "B6tHHS4UN2ix_OvlWXKFKHZXHOZM9K_SQoACTZRMeVU"
_acme-challenge.kinescopecdn.net. 120 IN TXT "yyYv3sCWKugtHGnprSeI_rZhW4_Z8HV4G8QWkv0F0iA"
_acme-challenge.kinescopecdn.net. 120 IN TXT "5T069_eJVXZy_LNZTx4B9imWZgGM-a8cedoMfSTExQk"
_acme-challenge.kinescopecdn.net. 120 IN TXT "H4KW2vE-U8YPQdrUz_LmQxr4l3cRcf4pgWg6HnTkYwc"

;; Query time: 52 msec
;; SERVER: 193.238.46.22#53(193.238.46.22)
;; WHEN: Thu May 06 13:46:43 CEST 2021
;; MSG SIZE rcvd: 1181

But Unboundtest - https://unboundtest.com/m/TXT/_acme-challenge.kinescopecdn.net/6JLQE4SN

Query results for TXT _acme-challenge.kinescopecdn.net

Response:
;; opcode: QUERY, status: NOERROR, id: 34615
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;_acme-challenge.kinescopecdn.net. IN TXT

;; AUTHORITY SECTION:
kinescopecdn.net. 0 IN SOA ns1.kinedns.net. noc.kinescope.io. 2021041501 7200 3600 1209600 3600

has an empty result, the output is too big.

Your code is buggy. Creating a certificate with *.example.com + example.com, two TXT entries are required.

Why there are more then 2 results?

2 Likes

The amount of records is not very important, we just made it as a sample proof of entries visibility. Anyway, we did leave only 2 records now:

dig TXT _acme-challenge.kinescopecdn.net.

; <<>> DiG 9.10.6 <<>> TXT _acme-challenge.kinescopecdn.net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50408
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.kinescopecdn.net. IN    TXT

;; ANSWER SECTION:
_acme-challenge.kinescopecdn.net. 120 IN TXT    "H4KW2vE-U8YPQdrUz_LmQxr4l3cRcf4pgWg6HnTkYwc"
_acme-challenge.kinescopecdn.net. 120 IN TXT    "5T069_eJVXZy_LNZTx4B9imWZgGM-a8cedoMfSTExQk"

;; Query time: 114 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri May 07 14:17:45 EEST 2021
;; MSG SIZE  rcvd: 173

We do dig from the local machine and see it in logs

May 07 11:17:45 kinescope-application-2 coredns[378]: [INFO] plugin/letsencrypt: _acme-challenge.kinescopecdn.net.: type=TXT, zone=kinescopecdn.net.

We don't see any requests from unboundtest in our logs though

Here's an unbound test: https://unboundtest.com/m/TXT/_acme-challenge.kinescopecdn.net/IAELJOML

I can confirm I see those TXT records using dig from my machine with dig -t TXT _acme-challenge.kinescopecdn.net

Let's Encrypt will use your authoritative nameserver and will check them in a few ways to ensure they all contain the required TXT record response.

I would recommend checking out acme-dns (CNAME redirection to a least privileged dns service)

2 Likes

Hmm... yet another domain with wildcard A records... :thinking:

Screenshot_20210507-060806_Samsung Internet

1 Like

By the way, you want an extremely short TTL on your TXT records and you want those to propagate to all nameservers as soon as possible. If there's any chance the results will be cached then validation for some combinations are more likely to fail.

In particular if you request a cert for domain.com and *.domain.com this results in a different TXT record value for the same record name and validation will fail if you don't present both those values when you attempt the challenge response, you will then need to create a whole new challenge response for whichever variation failed. Automation is the key but it must be timely. I've spent the last few weeks building a variation of acme-dns that theoretically scales to 10,000 domains per second :slight_smile:

2 Likes

Hi, @webprofusion we did a test with variable TTL time (shot and long) and had the same experience with Certbot. Is there any option to find out where Certbot query DNS servers are located and if there are some trust lists or whatever that prevent it from reaching our DNS servers?

1 Like

The online check https://check-your-website.server-daten.de/?q=kinescopecdn.net has seen your TXT records, Unboundtest doesn't - https://unboundtest.com/m/TXT/_acme-challenge.kinescopecdn.net/HBPK2NAW

So you must have something that blocks. What? No idea.

PS:

where Certbot query DNS servers are located

It's not Certbot, that's your ACME-client. It's the check of Letsencrypt.

PPS: Letsencrypt checks always the authoritative name servers, so it's not a problem of a wrong name server caching. But that produces some checks, that may hit a firewall or a ddos detection.

3 Likes

Your DNS server doesn't handle randomized capitalization of the name, which is used by unbound to mitigate some DNS attacks. (This is sometimes called DNS 0x20.)

$ dig +short +norecurse +bufsize=512 TXT _acme-challenge.kinescopecdn.net. @193.238.46.22
"H4KW2vE-U8YPQdrUz_LmQxr4l3cRcf4pgWg6HnTkYwc"
"5T069_eJVXZy_LNZTx4B9imWZgGM-a8cedoMfSTExQk"
$ dig +short +norecurse +bufsize=512 TXT _Acme-challenge.kinescopecdn.net. @193.238.46.22

(no result from the last command)

Your authoritative name servers need to support the case-insensitivity that DNS is supposed to have.

3 Likes

That's curious. "check-your-website" checks that ("Good: Nameserver supports Echo Capitalization: 2 good Nameserver").

But there was no error listed - https://check-your-website.server-daten.de/?q=kinescopecdn.net#comments

Other domains with buggy name servers -> that problem was listed.

Checked the log manual - there is no error.

Query "wWw.kInEsCoPeCdN.NeT", the result is "wWw.kInEsCoPeCdN.NeT".

But using dig, it's wrong:

:~$ dig TXT wWw.kInEsCoPeCdN.NeT. @ns2.kinedns.net.

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> TXT wWw.kInEsCoPeCdN.NeT. @ns2.kinedns.net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23189
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 4a6b5de98a389e20 (echoed)
;; QUESTION SECTION:
;wWw.kInEsCoPeCdN.NeT. IN TXT

;; AUTHORITY SECTION:
kinescopecdn.net. 3600 IN SOA ns1.kinedns.net. noc.kinescope.io. 2021041501 7200 3600 1209600 3600

;; Query time: 52 msec
;; SERVER: 193.238.46.23#53(193.238.46.23)
;; WHEN: Fri May 07 23:13:45 CEST 2021
;; MSG SIZE rcvd: 144

Checking my own domain, the result is correct:

:~$ dig TXT wWw.SeRvEr-dAtEn.De. @ns.inwx.de.

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> TXT wWw.SeRvEr-dAtEn.De. @ns.inwx.de.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56487
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1400
;; QUESTION SECTION:
;wWw.SeRvEr-dAtEn.De. IN TXT

;; AUTHORITY SECTION:
SeRvEr-dAtEn.De. 3600 IN SOA ns.inwx.De. hostmaster.inwx.De. 2021043000 10800 3600 604800 3600

;; Query time: 17 msec
;; SERVER: 2001:67c:1bc::104#53(2001:67c:1bc::104)
;; WHEN: Fri May 07 23:14:27 CEST 2021
;; MSG SIZE rcvd: 103

Question = Answer, not Lower(question) = answer.

If the same ip address has an authoritative and a recursive name server, such a problem may happen.

Sometimes the authoritative answers (correct), sometimes the recursive answers -> Echocapitalization-check will fail.

3 Likes

@JuergenAuer

I've looked at your two dig outputs for a few minutes now, but I can't seem to spot a difference other than the presence of the cookie line in the first output. What am I supposed to be looking at here? I feel like I'm missing something.

update:

oh, it's in the AUTHORITY section.

1 Like

In this case, I think the problem is worse than just the capitalization not echoing back (that is, asking for a name would give you the lowercased name in the answer). Here, instead it's that a not-lowercased name gives a NOERROR empty-result. The server is case-sensitive, rather than not-case-echoing.

My understanding from briefly looking into it is that not-case-echoing is maybe-technically-spec-compliant-but-unusual-and-causes-problems-with-DNS-0x20, while being case-sensitive is Just Plain Wrong. But the DNS standards can be really arcane, so I may not quite be understanding.

2 Likes

Ah yes, I had this problem last week. You need to reply with the same name in the response as they pass in the query e.g. _AcMe-Challenge. Just returning the correct TXT result isn't enough.

2 Likes

That's the 0x20-required case-echoing. But here, it's even worse: these servers don't even reply with the record at all if the query isn't lowercase.

RFC 1034 section 3.5 says,

Note that while upper and lower case letters are allowed in domain names, no significance is attached to the case. That is, two names with the same spelling but different case are to be treated as if identical.

But this server does not treat them identically.

2 Likes

Yes, there are two different problems.

The general idea of 0x20 is visible in that output:

;; QUESTION SECTION:
;wWw.SeRvEr-dAtEn.De. IN TXT

;; AUTHORITY SECTION:
SeRvEr-dAtEn.De. 3600 IN SOA ns.inwx.De. hostmaster.inwx.De. 2021043000 10800 3600 604800 3600

DNS uses a built-in-compression. A domain name is splitted into parts. Every part starts with one or two bytes. If the byte is lower 64, it's the length of the following character string. If the byte is 0x00, that's the end of the domain name. If the byte contains (bitwise) 128, the lower part + the next byte in combination defines the offset of the answer. There must start a part that ends with a 0x00.

Result: The query result nsinwx.De has one uppercase D, because the compression of (and the jump to) wWw.SeRvEr-dAtEn.De is used.

3 Likes

Hey, Christopher

Thank you (and all the guys in that thread ofc) for the valuable help that led us to the solution. :heart:
Indeed the issue was related to that requests were sent with qname in a different register.
Right after we fixed that we've been able to verify the cert:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/kinescopecdn.net-0001/fullchain.pem

For everyone who just participated in this thread, feel free to contact me in case you need any help with video hosting, streaming and related topics - free credits from our team are guaranteed :+1: We highly appreciate your help

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.