Domain authentication fails with dns-rfc2136 plugin

Help
1

My domain is: penguinpee.nl

I ran this command: certbot -v run -n -a dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/penguinpee.ini --agree-tos -m mailbox@penguinpee.nl --cert-name wildcard.penguinpee.nl -i nginx -d *.penguinpee.nl

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer nginx
Requesting a certificate for *.penguinpee.nl
Performing the following challenges:
dns-01 challenge for penguinpee.nl
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Challenge failed for domain penguinpee.nl
dns-01 challenge for penguinpee.nl

Certbot failed to authenticate some domains (authenticator: dns-rfc2136). The Certificate Authority reported these problems:
  Domain: penguinpee.nl
  Type:   dns
  Detail: During secondary validation: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.penguinpee.nl - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the DNS TXT records created by --dns-rfc2136. Ensure the above domains are hosted by this DNS provider, or try increasing --dns-rfc2136-propagation-seconds (currently 60 seconds).

Cleaning up challenges

My web server is (include version): nginx 1.20.2

The operating system my web server runs on is (include version): Fedora 35 (armv7l)

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.27.0

During the 60 second wait I am able to query named with dig _acme-challenge.penguinpee.nl +dnssec IN TXT and it provides an answer. Log entries also confirm that dns-rfc2136 plugin is able to add and remove the entry. I can also see queries coming in from Let's Encrypt servers for the specific records. Yet, in the end it fails.

In the log file, I found the following snippet:

{
  "type": "dns-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:dns",
    "detail": "During secondary validation: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.penguinpee.nl - check that a DNS record exists for this domain",
    "status": 400
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/123576657316/eWIHrw",
  "token": "rkl-fYjLqqHGYyKLdhwE7g4PQv75qt2EsAAtBflogXE",
  "validationRecord": [
    {
      "hostname": "penguinpee.nl"
    }
  ],
  "validated": "2022-06-25T16:59:28Z"
}

Are the secondary name servers also queried? In that case I would need to increase the propagation time considerably. My registrar is doing AXFRs once an hour and doesn't support NOTIFY. I'm not keen on lowering the TTL just for certificate updates.

1 Like
2

Let's Encrypt queries from multiple vantage points and each validation server chooses at least one nameserver randomly.

So yes, there's a high chance that DNS queries hit more than a single nameserver. One should also note that, when viewed from the outside, all nameservers are equal and there's no distinction between "primary" or "secondary" nameservers (even though it may be managed internally in this way).

You can delegate the _acme-challenge TXT record to a different zone using CNAMEs. This other zone can have a completly different DNS setup (for example that zone could have only a single nameserver) that will only be used for acme-challenge validation.

12 Likes
3

That's sounds a bit like what the dns-standalone plugin does. But does that work with dns-rfc2136?

I know I can put a CNAME in my zone for _acme-challenge pointing it to another zone (subdomain most likely). But how would I tell dns-rfc2136 to update that zone. That would require named to listen on a separate port at least, just for that zone, wouldn't it? Doable, but sounds a bit convoluted.

I don't mean to reject your suggestion. I'm just openly pondering my options here. There might be some value in having a delegated subdomain. It would limit the scope of what certbot's key is allowed to modify in BIND. :thinking:

4 Likes
4

It must provide that same answer from all four authoritative nameservers [probably where this fails - is 60 seconds enough time for them all to sync?]:

penguinpee.nl   nameserver = ns.penguinpee.nl
penguinpee.nl   nameserver = ns0.transip.net
penguinpee.nl   nameserver = ns1.transip.nl
penguinpee.nl   nameserver = ns2.transip.eu

which must be accessible from anywhere on the Internet [probably already the case].

9 Likes
5

No, it certainly isn't enough. As stated above:

1 Like
6

Yes, there are numerous ways for that. For example there's also acme-dns and much more. You can utilize a different nameserver software for the delegated zone and use that with either a plain, self-written auth-hook or utilize some existing plugins.

I'm not entirely sure, have never tried that. This depends on whether the plugin correctly follows the CNAME and adds the TXT record to the CNAME target. Had a quick look at the source code, looks like it depends on whether the SOA query follows the CNAME. Don't know if that's the case.

DNS nameservers must run on port 53, sadly. Unless you can do some magic DNS routing in your setup, you will either have to use the same nameserver that's already running on port 53 (any common nameserver software should be able to host multiple zones), or use a different machine or IP address (Let's Encrypt supports IPv6-only!) to host a different nameserver.

You can also consider hosting the delegated zone somewhere completly offsite (Cloudflare or something) and use their integrations for your ACME client, if you're willing to give up some authority.

This is actually one of the key benefits of the CNAME delegation. It can enable much better security, as your certbot plugin does not need access to your main DNS zone.

10 Likes
7

Sorry, I missed that.
If you control ns.penguinpee.nl, you could CNAME the requested FQDN to an FQDN that is only served by your single nameserver.
Like:
_acme-challenge CNAME local-stub.penguinpee.nl.
Where local-stub is a delegated subdomain and it is only being served by ns.penguinpee.nl.

10 Likes
8

I wasn't thinking about having named listen on a different port for outside queries. But for the RFC2136 zone update to work, the only parameters I can tweak are the IP address and the port that dns-rfc2136 plugin will use for performing the update.

I was thinking about using a different port for the delegated zone and tell dns-rfc2136 to use that port. But thinking some more about it, I'm not sure it's possible. I'm also not sure, yet, how dns-rfc2136 determines the FQDN it needs to update. That would need to match the domain of the delegated zone, but there's no parameter to tell it to update certs.penguinpee.nl instead of penguinpee.nl.

No worries.

Yes, that's basically what @Nummer378 suggested. But I'm still trying to find out if that's at all possible (see above).

Thank you both for your suggestions. I'll do some digging and report back tomorrow.

3 Likes
9

It looks like the suggested internal redirect to a delegated subdomain using a CNAME record doesn't work or I couldn't figure out a way to set this up.

Here is what I did and the results I got:

First attempt

I added _acme-challenge as a CNAME to my domain pointing it to certs.penguinpee.nl. I set up certs.penguinpee.nl as a delegated subdomain with its own SOA and a single NS entry pointing to the IP addresses (IPv4 and IPv6) where my local DNS server can be queried.

In BIND I configured allow-updates for certs.penguinpee.nl with the key assigned to certbots. When I run certbots I can observe the following queries in the log file:

query: _acme-challenge.penguinpee.nl IN SOA -T
query: penguinpee.nl IN SOA -T

Then it tries to update 'penguinpee.nl/IN', which is denied since the key is allowd only for certs.penguinpee.nl.

When I query for _acme-challenge.penguinpee.nl in either TXT or SOA the server returns the CNAME record pointing to certs.penguinpee.nl.

Second attempt

Same as above, except I changed the CNAME to point to _acme-challenge.certs.penguinpee.nl, hoping this would tell dns-rfc2136 what record to add and where. It didn't. Results where the same.

Third attempt

Since dns-rfc2136's first query is for a SOA record of _acme-challenge.penguinpee.nl I also tried changing certs.penguinpee.nl to _acme-challenge.penguinpee.nl as the name of the delegated zone. However, since _ is not an allowed character in hostnames (anything that resolves to an IP address), named threw multiple errors and warnings when checking the configuration.

Looks like I'm out of options using dns-rfc2136 other than setting --dns-rfc2136-propagation-seconds to 3600. :slightly_frowning_face:

1 Like
10

Digging further I found a related issue on GitHub. While the issue reported is for the PowerDNS plugin in the discussion further down it is mentioned that most DNS plugins behave the same. The suggested change is not yet merged, however.

Seeing that the OP described using an _acme-challenge subdomain, I was wondering why that wasn't working for me as described above. So, searched for the error messages I got and pulled out the BIND ARM and found the check-names option may need to be tuned to allow me to setup _acme-challenge.penguinpee.nl.

Stay tuned...

2 Likes
11

Alright, I nailed it and learned quite a bit along the way...

First of all BIND does support _acme-challenge as a delegated subdomain without modifying check-names. So, if named-checkconf complains about bad names there is something else wrong in the setup, that needs to be fixed, as was the case in my attempt.

That's a hint, which I failed to recognize myself until it finally dawned on me. You need to make sure there's nothing in the zone that references _acme-challenge. So, none of the common A, AAAA, NS, MX records. They all need to follow the rules for hostnames. In my case it was the NS records. But I failed to read the output properly.

In the end I added an NS record for _acme-challenge to penguinpee.nl instead of a CNAME. I also had to wait for that change to propagate to all secondary servers before I could request the certificate.

I think a CNAME would have worked as well. But maybe using NS, at least in my case, is a cleaner approach. That's up for debate. I'd be interested in hearing other people's opinion on that.

Obviously, my knowledge with respect to zone delegation isn't quite up to date which led to mistakes.

Thanks for the help and suggestions. :handshake:

2 Likes
12

Just to clarify, the linked issue does not apply to the dns-rfc2136 plugin. That works for a delegated _acme-challenge zone already.

2 Likes
13

There is a slight difference between making the controlled DNS zone the actual _acme-challenge FQDN and with using a CNAME to send that FQDN to another "less complicated" FQDN [which is then controlled by the single NS].
YMMV, but I would think that the second approach would be more compatible across DNS software and general/all purpose situations.
And probably less prone to failure; As the first delegation is via a simple CNAME. The second delegation can be done on a completely different system [and hopefully by the more experienced DNS admin - presuming situations where this is being done by more than one person / in more than one place].

9 Likes
14

I get your point with respect to preferring the simpler approach. But I might be missing something here in how that works out when using the dns-rfc2136 plugin.

Departing from what you suggested earlier:

Instead of setting up _acme-challenge I would have set up local-stub. Then I ask certbot to request a wildcard certificate for *.penguinpee.nl using the dns-rfc2136 authenticator. From the queries I could see in the log files, dns-rfc2136 was requesting SOA records for _acme-challenge.penguinpee.nl and failing to get an answer it moved up the chain, requesting the SOA record for penguinpee.nl. Since the first is a fail it tries to update penguinpee.nl with _acme-challenge IN TXT.

It doen't do a recursive query for _acme-challenge.penguinpee.nl. Otherwise my first attempt should have worked, right? But there it aso tried to update penguinpee.nl with a TXT record instead of certs.penguinpee.nl, the zone the CNAME pointed to.

1 Like
15

Ok, the "trick" seems to be in telling the client that FQDN-A is can be updated via FQDN-B/ZONE.
If that can't be done, then all bets are off [we lose].

13 Likes
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.