Secondary validation returns intermittent DNS SERVFAIL for survey.lang-sheng.cn

Hello,

I am requesting help diagnosing repeated secondary-validation DNS failures for a production HTTP-01 order.

Environment

  • Domain: survey.lang-sheng.cn
  • Expected A record: 8.138.25.158
  • Authoritative nameservers: dns1.hichina.com, dns2.hichina.com
  • Client: Certbot 1.22.0 with the Nginx plugin
  • Nginx: 1.24.0
  • OS: Alibaba Cloud Linux 3
  • Validation method: HTTP-01 through a dedicated virtual host on port 80

No certificate has been issued and I have stopped automatic retries.

Failure timeline (UTC+8, 2026-08-24)

  1. 16:53 — Account registration succeeded and the order started. Secondary validation timed out connecting to 8.138.25.158:80.
  2. 17:00 — A controlled retry failed because the CA resolver could not obtain the .cn DNSKEY (DNSKEY Missing / maximum sends exceeded).
  3. 17:35 — After a 30-minute quiet period and fresh DNS/HTTP checks, the final controlled retry failed with:
During secondary validation: DNS problem: SERVFAIL looking up A for survey.lang-sheng.cn;
DNS problem: SERVFAIL looking up AAAA for survey.lang-sheng.cn

The command was:

certbot --nginx --domain survey.lang-sheng.cn --non-interactive --redirect

Fresh independent checks before the final retry

  • Google DNS and Cloudflare DNS both returned status 0 for A with 8.138.25.158.
  • Both returned status 0 for AAAA with an empty answer and the HiChina SOA in authority data.
  • Both returned status 0 for the two HiChina authoritative nameservers.
  • Both returned the .cn DNSKEY with authenticated data (AD=true).
  • Ten out of ten independent Check-Host HTTP nodes returned 200 from 8.138.25.158 for http://survey.lang-sheng.cn/api/health.
  • The server returned 200 for both the loopback Host-routed health check and the public HTTP health check.
  • Nginx syntax passed and port 80 was listening on IPv4 and IPv6.
  • The public Let's Encrypt status page reported all systems operational at the time.

Post-failure state

  • /etc/letsencrypt/live/survey.lang-sheng.cn does not exist.
  • Port 443 is not listening.
  • The two approved Nginx configuration hashes are unchanged and nginx -t passes.
  • The application remains healthy on 127.0.0.1:3180, restart count 0, and public HTTP remains 200.
  • No DNS or firewall change was made.

Could someone help identify why the secondary validation path intermittently cannot resolve this .cn name while independent validating resolvers consistently return NOERROR? In particular, did the secondary resolver record a parent-zone DNSKEY timeout or another DNSSEC-validation failure before returning SERVFAIL for the child A/AAAA queries?

I will not retry production issuance again until there is a concrete explanation or the affected resolver path is known to be healthy. Thank you.

Welcome @mengzaiya520-spec Thank you for the detailed report. Others with more DNS experience than me may be able to help better. But, I wanted to at least get some response to you. And, give what may be helpful info.

You say "intermittently" get a secondary validation error. Is that because the other failures involve the timeout you describe? I don't see any history of a cert being issued for that domain so it looks like every attempt has failed.

Let's Encrypt chooses a random path as it walks the authoritative tree from top to bottom. In general, any path it chooses must succeed for the challenge to be successful. Looking at the DNSViz report shows many connectivity problems at the higher level DNS servers. I don't know what you can do about those other than report them to those operators. See: survey.lang-sheng.cn | DNSViz

These connectivity problems are almost certainly the reason for the timeout for your DNSKEY Missing error. It may also be causing the SERVFAIL but these can also be other things. Those can sometimes be very difficult to determine.

Walking the tree querying the authoritative DNS servers directly is different than the other resolvers you tried. Because of their overall design they are more tolerant of certain kinds of errors.

That is good to avoid failures to production. There are rate limits which will temporarily block you after too many consecutive failures in an hour.

You can test with the Let's Encrypt Staging system though. It is less strict on rate limits. Note the --nginx plugin does not support that directly but a slight modification to your command will use Staging for a test. A successful staging test will not update your nginx config and will not store a certificate. But, it is good for testing DNS queries and challenges.

Try this command

certbot certonly --nginx --dry-run -d survey.lang-sheng.cn 

It would be interesting to know the result of this command. Even try multiple times to see if you get consistent results.

It is possible there is some issue in one or more of the LE secondary centers. But, with such poor connectivity results for your entire DNS tree it is very hard to debug. We are not seeing other reports of a similar problem. At this stage this looks related to your DNS configuration rather than LE. As I noted earlier, perhaps others with more DNS experience will have more to say.

Hi @MikeMcQ,

Thank you for the suggestions. I ran the staging dry-run test as you advised — 5 consecutive attempts of certbot certonly --nginx --dry-run -d survey.lang-sheng.cn from the production server (Aliyun lightweight server, Guangzhou region). All 5 runs failed with the SAME error, and notably this is NOT a DNS error:

requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

Each run times out at the HTTPS read stage when talking to the Let's Encrypt staging ACME API. This looks like an egress connectivity issue between this Aliyun Guangzhou server and the LE endpoints — separate from the earlier secondary-validation SERVFAIL, which happened in the opposite direction (LE resolver → HiChina nameserver).

For reference, the earlier production failures were:

  • 17:00 CST Aug 24: .cn DNSKEY Missing / maximum sends exceeded
  • 17:35 CST Aug 24: During secondary validation: DNS problem: SERVFAIL looking up A / AAAA for survey.lang-sheng.cn

Additional local evidence from Aug 24: the authoritative nameservers dns1/dns2.hichina.com advertise AAAA 2408:4009:501::9 / ::10, but TCP 53 to those IPv6 addresses is unreachable from our network while the IPv4 path (120.76.107.43 / 39.96.153.43) is open — consistent with the pattern you described in topic 181224.

So it seems we may have two independent problems: (1) LE secondary validation toward the .cn nameservers, and (2) this server's egress path to the LE ACME API. We will not retry production issuance until the egress path is stable. Happy to provide the full dry-run output if useful.

Yes, I agree.

It is not LE resolver only to HiChina name server. LE starts at the root servers and works down the tree. The DNSViz link I provided shows a nice picture of this. It also shows many connectivity problems for the DNS servers above HiChina. It is these higher level DNS servers that are contributing to your problem. They may even be the cause of all of them.

same issue with CN domain :frowning:

in that case, not really have proof but I feel like LE's secondary watchpoint got IP that banned from GFW? IIRC they use AWS for secondary observer point so

I encountered the same problem. During secondary validation: While processing CAA for mydomain.cn: DNS issue: query timed out while looking up CAA for mydomain.cn When will it be resolved?

@crystalsu2001 It's possible you're running into a separate issue; you may want to open your own thread, using your actual domain name.

However, the limited evidence I can see is that the .cn TLD nameservers might no longer be replying to requests for everyone worldwide. You need to have working DNS servers worldwide in order to prove control over a domain worldwide in order to get a valid-worldwide certificate. Let's Encrypt checks from multiple places in order to help confirm that a requester is actually in control of the name and that it's not an attacker who can only impact some of the Internet. So if your domain name isn't working (from an external perspective), you won't be able to get a certificate, from Let's Encrypt or from anyone else that the authoritative DNS servers are blocking.

Some online tools that you might be able to use to see if your DNS servers are working externally include:

But those only test if your DNS servers are accessible from where that test system is on the Internet, not from everywhere like a Certificate Authority has to do.

If you're expecting that your domain name should be accessible worldwide, and it looks like it isn't, then I recommend contacting whomever it is that you pay for DNS service to look into it.