Cert-manager + dns01 + renewal + BIND w/ wildcard, causes dns-downtime

Help
1

My domain is:
upperpaste.com

I can login to a root shell on my machine (yes or no, or I don't know):
yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Cert-manager

I have a rather curious problem. I've been running the following for quite a while now

Bind9 with a zone like:

...
$ORIGIN upperpaste.com.
*                       CNAME   dd.home
dd.home          A              1.2.3.4
...

On my kubernetes cluster I installed cert-manager using rfc2136 with above bind/zone, for at least one subdomain like, objects.upperpaste.com. Note, I rely on the dns wildcard above to resolve objects.upperpaste.com

Suddenly most of my subdomains were not working anymore. Investigation showed that due to an unrelated problem, all _acme-challenges were in my zone file and were not removed. However because there were now resource records like,

_acme-challenge.objects.corp.worldstream.com TXT ...

my dns wildcard no longer hit. (which makes sense from BIND's perspective)

My question is, if you can't/won't make any subdomain A/CNAME records, but do want to have reliable DNS01 ACME challenging, without it possibly invalidating wildcards during challenges (just like in my above case), how would you proceed? *
I've started using external-dns1, but it seems heavy-handed just to remediate my above problem, in my opinion the wildcard should just stay working.

* you could argue that in a correct setting, the _acme-challenge TXT record will only be there for mere seconds, however I think that still unacceptable as it leaves a window in which dns-clients might wrongfully get an NXDOMAIN for the intended subdomain which happens to be under ACME DNS01 challenge.

PS
I think this is not intrinsic for cert-manager, but actually a more general problem to DNS01, that's why I posted here first, before asking at cert-manager's forums

2

Hi Hans, and welcome to the LE community forum.

I can't say that I understand the problem (as explained).
[foremost, I can't find where the CNAME is being used]
But I have found this information which may be useful for you to resolve the problem:

Name:       objects.upperpaste.com
Addresses:  192.168.1.222
            192.168.1.156
            192.168.1.203


Last of which shows that, although your domain lists two nameservers - they both show the same IPs, and the IPv6 address is not responding to DNS queries:

image
image1046×359 22.2 KB

[debatable] LE prefers IPv6 when present and this may be the root cause of your problem.
You should [I would] either correct the IPv6 DNS problem, or remove the IPv6 address from the names before continuing attempts at issuing new certs.

1 Like
3

Nice find with the non-working IPv6 nameservers, definitely worth fixing.

I think things work a little bit differently in this respect if we're talking about Unbound (vs Boulder). Unbound is probably pretty robustt robust against IPv6 problems like this. At least, I don't see any evidence in the OP's post that point to this as causing problems.

Yeah, the wildcard algorithm is hurting you pretty badly here:

Wildcard RRs do not apply:

  • ...
  • When the query name or a name between the wildcard domain and
    the query name is know to exist. For example, if a wildcard
    RR has an owner name of "*.X", and the zone also contains RRs
    attached to B.X, the wildcards would apply to queries for name
    Z.X (presuming there is no explicit information for Z.X), but
    not to B.X, A.B.X, or X.

I can't think of a good solution either.

What's more, I bet there's probably a handful of users out there who are being burned by this and not even realizing it!

2 Likes
4

Yeah, the current mess of my Dns it not really representative of the problem described in the OP.

@_az seems to get it though.

I can only summerize as

  1. You depend on a DNS wildcard in your DNS server for proper working of your subdomain (not to be confused with a wildcard certificate)
  2. A DNS challenge starts as such the automation, in my case rfc2136, adds a _acme-challenge.subdomain to my zone file.
  3. The DNS server whilst having the _acme-challenge.subdomain.example.com causes an NXDOMAIN for subdomain.example.com, since from the DNS servers perspective the wildcard is no longer matched for subdomain.example.com.

The longer the window of step 2., the more NXDOMAINs will be served by the DNS server for the subdomain being challenged.

5

Can you use DNS delegation?
[do you have more than one Internet IP - more than one DNS server]
A working solution may require adding an additional subdomain zone.
Just thinking through this...

6

I only have one static IP.

I'm really interested in where you're trying to go with this :slight_smile:

7

It seems to be looping (in my head) ...
You seem to need a CNAME of a wildcard TXT record:
* TXT CNAME other.zone
Which is clearly impossible to do.
But if we try to break that down somehow...

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.