Before cloudflare fix the problem, I suggest:
zone "auth.acme-dns.io" IN {
type forward;
forward only;
forwarders { 46.4.128.227; 9.9.9.9; };
};
OR
/etc/resolve.conf
nameserver 9.9.9.9
This is what I can do at client side.
1 Like
Hi there,
Let's Encrypt does not use third-party public DNS resolvers to conduct validation. (We use Cloudflare as the front-end Content Delivery Network [CDN] for our API, but we do not use their resolvers.)
We run our own DNS resolvers, which start with the root nameservers and follow delegations from there. As long as your DNS is set up correctly, then we should be able to resolve it.
So, in order to resolve your problem, we would need to do fundamental DNS troubleshooting. For example:
- When you added the
_acme-challengerecord, are you certain it had fully propagated to every one of the authoritative DNS servers forcomp.hkbu.edu.hkbefore you ran Certbot? - Let's Encrypt has a (very short) DNS cache. If you noticed a problem with your DNS record(s) and corrected it, did you re-run Certbot immediately, or did you wait a few minutes to allow a missing/incorrect record to expire from our cache?
It looks like troubleshooting is a little more complicated because you're using acme-dns.io; right now, _acme-challenge.comp.hkbu.edu.hk is a CNAME record pointing to 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io, which does not have any records. So, similar questions apply to that service:
- Is the
acme-dns.ioservice still working and is your account there still active? - If that TXT record is created on the fly by a plugin, is the plugin making sure that
acme-dns.iohas published the record before Certbot contacts Let's Encrypt to finish the certificate request?
Troubleshooting is a little trickier still because a lot of "convenient" DNS troubleshooting tools assume that your goal is to have something that works like an A record, not "just" a TXT record, and so their output might be confusing.
I'm puzzled why Quad9 and Unboundtest can resolve those acme-dns.io TXT records, but Let's Encrypt and my own personal resolvers cannot. @joohoi, any insight?
4 Likes
The problem is resolving: 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io, not resolving _acme-challenge.comp.hkbu.edu.hk which is cloudflare problem.
;; QUESTION SECTION:
;4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io. IN TXT
;; AUTHORITY SECTION:
acme-dns.io. 1799 IN SOA bonnie.ns.cloudflare.com. dns.cloudflare.com. 2034849170 10000 2400 604800 3600
I think that cloudflare as a global service provider, some DNS work, some DNS do not. Therefore, if LE lookup acme-dns.io directly from auth.acme-dns.io (46.4.128.227) or via Quad9, which is by-passing cloudflare, it works.
Hope this help.
1 Like
I think I'm starting to see what's going on. Sorry this is so verbose; it's late in my time zone and I'm puzzled.
Two of Cloudflare's authoritative nameservers (bonnie.ns.cloudflare.com and pablo.ns.cloudflare.com) are listed (with the .io registry) as the authoritative nameservers for acme-dns.io. So, a DNS resolver will query them as the first step in resolving 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io.
If you query bonnie or pablo directly, asking for 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io, they will say that it doesn't exist (NXDOMAIN).
However, if you query bonnie or pablo for auth.acme-dns.io, they will answer with a NS record delegating responsibility to ns.auth.acme-dns.io. And ns.auth.acme-dns.io does return results for 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io.
I agree that this is a Cloudflare bug: their nameservers should be returning that NS delegation whenever they are queried for a subdomain of auth.acme-dns.io.
I think the reason this is working for (or through) some resolvers, but not others, is that some resolvers use an optional feature called "QNAME minimisation." At the cost of having to make more queries (which is a little bit slower), they increase users' privacy by making a separate query for each level of a hostname that they're trying to resolve. For example, they would query bonnie for only auth.acme-dns.io first (to see if it's delegated elsewhere), instead of querying it for the full hostname 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io right away.
So, resolvers using QNAME minimisation will probably be able to resolve your TXT records, but resolvers not using QNAME minimisation will fail.
Ironically, QNAME minimisation has tended to cause problems with buggy authoritative servers, instead of fixing them. This is the first time I've seen it be the other way around.
Let's Encrypt's resolvers currently do not use QNAME minimisation.
I will open a support ticket with our Cloudflare contacts to let them know about this, since it looks like a bug on their side (even though it's not with our infrastructure).
To work around this and renew your certificate in the meantime, I recommend you temporarily add an _acme-challenge DNS record manually, in place of the CNAME you're using. Hopefully, the underlying problem will be fixed well before your next renewal.
Unfortunately, I don't think we can work around this on our side by turning on QNAME minimisation. We'd have to do some risk analysis before changing a setting like that, which is not something we can do quickly unless it's a grave emergency.
3 Likes
I have already had:
_acme-challenge in cname 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io.
What should I do for modification? I can't get your meaning. Thanks.
1 Like
Try removing the CNAME record and removing --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py from your command line. Certbot will give you TXT records that you can add to your DNS instead of the CNAME. That should work.
Once you've issued a certificate this way, Certbot won't be able to automatically renew. The next time renewal is coming up, you'll probably want to put the CNAME record back in place and use the manual-auth-hook again. After the problem is fixed and that next renewal is done, automatic renewal will work again in the future.
2 Likes
I see. For your reference, you may take my use-case to talk with cloudflare. My cert will be expired on 19 July 2021. I'll wait for the fix until 7-10 days expiry in advance. Will you expect cloudflare may fix it shortly?
Another problem for giving TXT record in my DNS (renew manually), I experienced that the lookup from LE is very slow and not fresh enough that causes the checking fails. I don't know how long LE side can learn the updated TXT. (how many seconds after updated?) If possible, each checking repeats more times (to be increased) until it throws rejection. Any suggestions?
Thanks.
2 Likes
As stated before, LE does not use 3rd party DNS resolvers and won't use them in the future, it's no use suggesting this over and over.
I'm pretty sure the only persons who can answer that is Cloudflare, not LE staff.
Thanks! I hope Cloudflare will fix it quickly, but I can't be sure; I might be wrong about the problem, and there might be some other problem that's specific to acme-dns.io that might or might not get fixed. We'll have to see.
We currently discard our cache every 60 seconds, so if you wait that long after publishing the updated TXT record, it should work. (N.b. This timing might change in the future, maybe without notice, so please don't rely on it.)
2 Likes
Just to confirm this seems to be a general problem with all attempts to use the hosted (@joohoi ) acme-dns service. The ACME service for ssl.com works, so they obviously resolve their DNS differently.
2 Likes
I suppose now might be as good a time as any to say my company is offering a new managed acme-dns compatible service via https://certifytheweb.com/
To try it out, create an account (sign in), then on your License Keys tab click Enable Certify DNS to activate a license key for the service. Once enabled a URL will be shown to use as the acme-dns service base URL (this is currently only compatible with basic authentication aware clients).
It's free for now and may or may not have a free tier in the future (it probably will and will always be inexpensive), but it's free now for folks who want to kick the tyres.
2 Likes
dig @8.8.8.8 _acme-challenge.comp.hkbu.edu.hk txt
;; QUESTION SECTION:
;_acme-challenge.comp.hkbu.edu.hk. IN TXT
;; ANSWER SECTION:
_acme-challenge.comp.hkbu.edu.hk. 21599 IN CNAME 4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io.
4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io. 0 IN TXT "yljK66p6R11dtfQTmdn_0YLXwBp0dRgNE2l1WBiNzK8"
4aac37e7-1c30-43f0-864c-26b888dac908.auth.acme-dns.io. 0 IN TXT "Ogp0LYabmAMvyNFve3xmmDDIrAgTYi0rraRPKpVYkPE"
I would like to reply you all that the auto renewal has been committed this morning. Thanks.
I have to express one more point in this thread. For those who (so-called community leader) always replied to general public that "LE has no fault. LE don't.... ", please listen to us carefully and try to answer with relatively helpful manner. I quite appreciate @JamesLE and @griffin for their attitude to study the problem.
2 Likes
It isn't always easy to hear the simple and plain truth.
I don't know that anything was written to be unhelpful and I'm glad you seem to understand how DNS validation is handled.
If not, here is a simple breakdown.
Where is xyz.domain.com.?
Step #1: Ask the "." (root) DNS servers where and they will reply with: Go ask the "com." DNS servers.
Step #2: Ask the "com." DNS servers where and they will reply with: Go ask the "domain.com." DNS.
Step #3: Ask the "domain.com." DNS servers and they should have the answer - unless it is a CNAME to some other name (if so, you take that name and begin with "." and work your way left until you have the answer).
No forwarders are ever used to insure that no one can manipulate the system.
1 Like
If you're referring to me, explaining that Let's Encrypt won't start using third party resolvers: I just didn't want you to get any false hope.
2 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.