DNS problem: SERVFAIL looking up CAA for tk

It looks a little bit like the timeouts are happening shortly after the records are created, and they they disappear. I caught some timeouts while you were adding the TXT records to dvolve.tk.

What if you put an extra long delay between updating the records and responding to the challenge?

I am reminded of another thread, not about DigitalOcean or Linode, but giving BIND more time to reload resolved some SERVFAILs.

At the moment, I’m doing a 4 minute delay between record creation and responding to the challenges which historically seems to be plenty for these two providers. But let me double it and see. Actually, first I’ll check if having the fake record there already will change anything.

I also still can’t really understand why this is only affecting the .tk domains.

P.S. The Linode zone is actually a slave from an on-prem master, so it updates way faster than a Linode primary.

Found a more reliable reproduction of the timeouts with non-existent domains. The more tk domains you have, the more likely one of them hits the timeout.

tl;dr; it’s a big mix of timeouts, NXDOMAIN and SERVFAIL. Just like you’ve been seeing.

I think the most likely explanation is that the tk TLD nameservers are rate limiting Let’s Encrypt.

Not sure who to tag, apparently some of the staff are traveling …

sudo certbot-auto certonly -a manual --preferred-challenges dns \
--dry-run -n \
--manual-auth-hook "/bin/true" --manual-cleanup-hook "/bin/true" \
--manual-public-ip-logging-ok \
-d fake-domain-1.tk \
-d fake-domain-2.tk \
-d fake-domain-3.tk \
-d fake-domain-4.tk \
-d fake-domain-5.tk \
-d fake-domain-6.tk \
-d fake-domain-7.tk \
-d fake-domain-8.tk \
-d fake-domain-9.tk \
-d fake-domain-10.tk \
-d fake-domain-11.tk \
-d fake-domain-12.tk \
-d fake-domain-13.tk \
-d fake-domain-14.tk \
-d fake-domain-15.tk \
-d fake-domain-16.tk \
-d fake-domain-17.tk \
-d fake-domain-18.tk \
-d fake-domain-19.tk \
-d fake-domain-20.tk \
-d fake-domain-21.tk \
-d fake-domain-22.tk \
-d fake-domain-23.tk \
-d fake-domain-24.tk \
-d fake-domain-25.tk \
-d fake-domain-26.tk \
-d fake-domain-27.tk \
-d fake-domain-28.tk \
-d fake-domain-29.tk \
-d fake-domain-30.tk \
-d fake-domain-31.tk \
-d fake-domain-32.tk \
-d fake-domain-33.tk \
-d fake-domain-34.tk \
-d fake-domain-35.tk \
-d fake-domain-36.tk \
-d fake-domain-37.tk \
-d fake-domain-38.tk \
-d fake-domain-39.tk \
-d fake-domain-40.tk \
-d fake-domain-41.tk \
-d fake-domain-42.tk \
-d fake-domain-43.tk \
-d fake-domain-44.tk \
-d fake-domain-45.tk \
-d fake-domain-46.tk \
-d fake-domain-47.tk \
-d fake-domain-48.tk \
-d fake-domain-49.tk \
-d fake-domain-50.tk \
-d fake-domain-51.tk \
-d fake-domain-52.tk \
-d fake-domain-53.tk \
-d fake-domain-54.tk \
-d fake-domain-55.tk \
-d fake-domain-56.tk \
-d fake-domain-57.tk \
-d fake-domain-58.tk \
-d fake-domain-59.tk \
-d fake-domain-60.tk \
-d fake-domain-61.tk \
-d fake-domain-62.tk \
-d fake-domain-63.tk \
-d fake-domain-64.tk \
-d fake-domain-65.tk \
-d fake-domain-66.tk \
-d fake-domain-67.tk \
-d fake-domain-68.tk \
-d fake-domain-69.tk \
-d fake-domain-70.tk \
-d fake-domain-71.tk \
-d fake-domain-72.tk \
-d fake-domain-73.tk \
-d fake-domain-74.tk \
-d fake-domain-75.tk \
-d fake-domain-76.tk \
-d fake-domain-77.tk \
-d fake-domain-78.tk \
-d fake-domain-79.tk \
-d fake-domain-80.tk \
-d fake-domain-81.tk \
-d fake-domain-82.tk \
-d fake-domain-83.tk \
-d fake-domain-84.tk \
-d fake-domain-85.tk \
-d fake-domain-86.tk \
-d fake-domain-87.tk \
-d fake-domain-88.tk \
-d fake-domain-89.tk \
-d fake-domain-90.tk \
-d fake-domain-91.tk \
-d fake-domain-92.tk \
-d fake-domain-93.tk \
-d fake-domain-94.tk \
-d fake-domain-95.tk \
-d fake-domain-96.tk \
-d fake-domain-97.tk \
-d fake-domain-98.tk \
-d fake-domain-99.tk \
-d fake-domain-100.tk
<snip>
IMPORTANT NOTES:
- The following errors were reported by the server:

  Domain: fake-domain-1.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-1.tk

  Domain: fake-domain-100.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-100.tk

  Domain: fake-domain-11.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-11.tk

  Domain: fake-domain-13.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-13.tk

  Domain: fake-domain-14.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-14.tk

  Domain: fake-domain-16.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-16.tk

  Domain: fake-domain-17.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-17.tk

  Domain: fake-domain-18.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-18.tk

  Domain: fake-domain-26.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-26.tk

  Domain: fake-domain-33.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-33.tk

  Domain: fake-domain-36.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-36.tk

  Domain: fake-domain-37.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-37.tk

  Domain: fake-domain-38.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-38.tk

  Domain: fake-domain-39.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-39.tk

  Domain: fake-domain-4.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-4.tk

  Domain: fake-domain-41.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-41.tk

  Domain: fake-domain-43.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-43.tk

  Domain: fake-domain-45.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-45.tk

  Domain: fake-domain-46.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-46.tk

  Domain: fake-domain-47.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-47.tk

  Domain: fake-domain-49.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-49.tk

  Domain: fake-domain-5.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-5.tk

  Domain: fake-domain-52.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-52.tk

  Domain: fake-domain-53.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-53.tk

  Domain: fake-domain-57.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-57.tk

  Domain: fake-domain-58.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-58.tk

  Domain: fake-domain-59.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-59.tk

  Domain: fake-domain-6.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-6.tk

  Domain: fake-domain-60.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-60.tk

  Domain: fake-domain-61.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-61.tk

  Domain: fake-domain-63.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-63.tk

  Domain: fake-domain-65.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-65.tk

  Domain: fake-domain-66.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-66.tk

  Domain: fake-domain-67.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-67.tk

  Domain: fake-domain-68.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-68.tk

  Domain: fake-domain-69.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-69.tk

  Domain: fake-domain-7.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-7.tk

  Domain: fake-domain-70.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-70.tk

  Domain: fake-domain-71.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-71.tk

  Domain: fake-domain-72.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-72.tk

  Domain: fake-domain-73.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-73.tk

  Domain: fake-domain-74.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-74.tk

  Domain: fake-domain-75.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-75.tk

  Domain: fake-domain-76.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-76.tk

  Domain: fake-domain-77.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-77.tk

  Domain: fake-domain-78.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-78.tk

  Domain: fake-domain-79.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-79.tk

  Domain: fake-domain-8.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-8.tk

  Domain: fake-domain-80.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-80.tk

  Domain: fake-domain-81.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-81.tk

  Domain: fake-domain-82.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-82.tk

  Domain: fake-domain-83.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-83.tk

  Domain: fake-domain-84.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-84.tk

  Domain: fake-domain-85.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-85.tk

  Domain: fake-domain-86.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-86.tk

  Domain: fake-domain-87.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-87.tk

  Domain: fake-domain-88.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-88.tk

  Domain: fake-domain-89.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-89.tk

  Domain: fake-domain-9.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-9.tk

  Domain: fake-domain-90.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-90.tk

  Domain: fake-domain-91.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-91.tk

  Domain: fake-domain-92.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-92.tk

  Domain: fake-domain-93.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-93.tk

  Domain: fake-domain-94.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-94.tk

  Domain: fake-domain-95.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-95.tk

  Domain: fake-domain-96.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-96.tk

  Domain: fake-domain-97.tk
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for
  _acme-challenge.fake-domain-97.tk

  Domain: fake-domain-98.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-98.tk

  Domain: fake-domain-10.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-10.tk

  Domain: fake-domain-12.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-12.tk

  Domain: fake-domain-15.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-15.tk

  Domain: fake-domain-19.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-19.tk

  Domain: fake-domain-2.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-2.tk

  Domain: fake-domain-20.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-20.tk

  Domain: fake-domain-21.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-21.tk

  Domain: fake-domain-22.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-22.tk

  Domain: fake-domain-23.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-23.tk

  Domain: fake-domain-24.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-24.tk

  Domain: fake-domain-25.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-25.tk

  Domain: fake-domain-27.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-27.tk

  Domain: fake-domain-28.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-28.tk

  Domain: fake-domain-29.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-29.tk

  Domain: fake-domain-3.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-3.tk

  Domain: fake-domain-30.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-30.tk

  Domain: fake-domain-31.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-31.tk

  Domain: fake-domain-32.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-32.tk

  Domain: fake-domain-34.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-34.tk

  Domain: fake-domain-35.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-35.tk

  Domain: fake-domain-40.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-40.tk

  Domain: fake-domain-42.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-42.tk

  Domain: fake-domain-44.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-44.tk

  Domain: fake-domain-48.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-48.tk

  Domain: fake-domain-50.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-50.tk

  Domain: fake-domain-51.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-51.tk

  Domain: fake-domain-54.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-54.tk

  Domain: fake-domain-55.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-55.tk

  Domain: fake-domain-56.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-56.tk

  Domain: fake-domain-62.tk
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for
  _acme-challenge.fake-domain-62.tk

  Domain: fake-domain-64.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-64.tk

  Domain: fake-domain-99.tk
  Type:   dns
  Detail: DNS problem: query timed out looking up TXT for
  _acme-challenge.fake-domain-99.tk

Yeah, my last test with the pre-existing fake record plus the 2 real records for each domain ended up with a couple SERVFAIL looking up CAA and a couple SERVFAIL looking up TXT.

Haha, that was probably while my 100 validations were in flight …

Yep, I think this is a correct diagnosis.

Yep, I also think this is the cause. We've seen similar cases before. We've also seen other strange issues from the tk TLD in the past, but I don't recall whether this was the specific problem.

Note that DNS rate limit problems may have gotten worse recently. We've had multi-perspective validation enabled in staging for quite a while now, and have seen a handful of reports of problems that seem to be related to DNS server rate limits. In the last couple of months, we also enabled multi-perspective validation in prod. We don't have it set to enforce mode yet, so failure of remote validations won't stop a prod issuance, but the extra queries triggered by remote validation could trigger aggressive rate limiters, which would then potentially affect queries by the core prod infrastructure.

I notice the original poster was using --dry-run, which would do a staging issuance. Are there any instances here where this is prevent renewal of a live cert?

Feel free as always to tag @lestaff if you're not sure! Thanks for all the analysis and thinking you've done in this thread so far.

1 Like

I haven't wanted to risk running afoul of prod rate limits in my testing.

For kicks, I went and found the .tk content policy which among other things mentions

The Dot TK Registry does not allow any free domain name registrations of which the referred
websites contain content similar as stated in the categories below:

NON-EXISTING PAGES – This category includes domains where the content does not exist,
shows an empty page, shows an error message, shows a ā€˜not available’ page or is hidden
behind a firewall or used with a VPN and is therefore not available for the public and for Dot
TK’s content verification systems.

I definitely don't have anything associated with dvolve.tk yet and I think poshacme.tk just ends up redirecting to my project page on GitHub. Wonder if my zones are flagged as being non-compliant with their "content verification systems".

I'm also curious if the potential rate limiting applies to the other free TLDs they maintain (.TK / .ML / .GA / .CF / .GQ).

There was a thread reporting SERVFAILs for .ga for the live ACME environment just two days ago, which I could reproduce at the time.

As jsha mentioned it seems like there’s a problem that needs to be solved with this new multi-VA stuff: one person can globally prevent any domain validation for any of those ccTLDs for a good 30 seconds or more with just one Let’s Encrypt order.

1 Like

I don’t think there’s a problem unique to Let’s Encrypt.

From two of my VPSes (in one ASN), I can’t access any of their nameservers over IPv6. If there’s rate limiting, either it banned me for 12+ hours after a handful of queries, or it’s applied to blocks larger than /64.

I haven’t checked today, but DNSViz reported the same thing yesterday. (Then again, DNSViz reports IPv6 problems so frequently that I don’t trust it.)

I also tried one or two of the quad N resolvers yesterday, and they were sometimes slow or failed to resolve .tk names.

2 Likes

Thinking more about the rate limit hypothesis, I realize it would have to be a global rate limit on responses for multi-va to have any affect. Otherwise, the secondary queries come from other IPs and would wind up in another rate limit bucket.

I remember someone ran into limiting issues not too long ago, running their own recursive resolver with limits locked way down. I can’t recall off the top of my head whether that was a global limit or not.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.