Production HTTP-01 validation times out immediately after staging succeeds (Primary validator only, no firewall block found)

Help
1

My domain is: test.compute.sta1.nscale.com (resolves to 193.143.78.6)

I ran this command (staging, succeeds):slight_smile: lego --email=notifications@example.com --domains=test.compute.sta1.nscale.com --server https://acme-staging-v02.api.letsencrypt.org/directory --http.webroot /etc/letsencrypt/http-01 --accept-tos --http run

(note: notifications@example.com is a placeholder - redacted our real account email for this post)

Then, ~18 seconds later, the same command against production (fails):slight_smile: lego --email=notifications@example.com --domains=test.compute.sta1.nscale.com --server https://acme-v02.api.letsencrypt.org/directory --http.webroot /etc/letsencrypt/http-01 --accept-tos --http run

It produced this output:
Staging: "The server validated our request" -> certificate issued successfully.

Production, immediately after, fails every time with:
acme: error: 400 :: urn:ietf:params:acme:error:connection :: 193.143.78.6: Fetching http://test.compute.sta1.nscale.com/.well-known/acme-challenge/[token]: Timeout during connect (likely firewall problem)

Two independent confirmed failures today (13 Aug), both UTC:

  • 10:42:13-10:42:36, token eY8jBlRmqK2X-djbOLMpWo5E_C7SBcFXcCtXzDen28I
    • 10:48:53-10:49:11, token HQrQFZ0NONJk0-sltgaq9hM2KNawm50htMsj059Tbis
      My web server is: Apache, serving the webroot directly (nothing unusual in vhost/DocumentRoot config - confirmed by staging succeeding with the identical setup seconds earlier).

The operating system my web server runs on is: Ubuntu 22.04 LTS (containerized ACME client via Docker).

My hosting provider, if applicable, is: self-hosted, private data center (OpenStack compute node), not a public cloud.

I can login to a root shell on my machine: yes.

I'm using a control panel to manage my site: no.

The version of my client is: lego (containerized).

Additional context, since this doesn't match a typical firewall-block pattern and we'd like to understand what's happening on your side specifically:

  • The error carries no "During secondary validation:" prefix, so it appears to be the Primary validator itself timing out, not a secondary perspective.
    • 193.143.78.6 is reachable from 25/25 diverse external check-host.net vantage points on TCP/80, and correctly serves a 404 for a bogus challenge token from 12/12 HTTP probes. It is not a general reachability problem.
    • During the successful staging run seconds earlier, our perimeter firewall logged all 5 validator connections cleanly (allowed, clean TCP close): source IPs 13.213.78.76 (Singapore), 66.133.109.36, 3.148.226.41, 13.63.62.195 (Sweden), 54.201.8.134 (all US except noted). This is the reference signature for genuine validator traffic reaching us successfully.
    • During both production failures, our firewall logs show zero connections from any of those IPs, or any other source, to port 80 on this address, in the exact failure windows. This isn't a case of "arrived and got blocked" - nothing arrives at all.
    • We've directly ruled out, on our own infrastructure: geoblocking (no source-region restriction on the allow rule, no relevant deny rules above it), flood/connection-rate protection (thresholds orders of magnitude above this traffic volume), packet buffer protection (enabled, but zero related events logged in either failure window), and asymmetric-path session handling (deliberately disabled for a direct test - failure persisted identically).
    • A live, fully-instrumented paired test (firewall logs, packet capture at our edge and at our host, tcpdump on the host) is planned to catch this happening again, but as of today it has not happened during an already-completed capture window - the failures happen in a matter of seconds and our capture point had to be relocated mid-test on one occasion.
      Given the above, we're trying to understand whether this could be an issue on Let's Encrypt's Primary validator's own outbound path (e.g. a peering/interconnect issue with the network(s) upstream of us), similar to a pattern described in an older community thread (Timeout during connect (likely firewall problem)) where a carrier-level interconnect problem was suspected. If you're able to look up either of the two failed authorizations above and tell us what your validator observed on its end (e.g. no response to SYN, a response that never completed, or something else), that would help us rule our own infrastructure in or out conclusively. Happy to provide any further detail.
2

Welcome @ssadiq Thanks for a terrific report. I agree that the most likely problem is some network routing problem between the LE primary center and your site.

You linked to a thread where I'd commented on a similar problem. And, I don't have a lot more to add than what I said there. But, I do have some ...

First, are you a customer of nscale or are you part of their support team?

Second, I see you are using Cloudflare as your DNS provider. You could switch to using a DNS Challenge instead. That avoids LE having to send HTTP requests for validation. Cloudflare is well-supported by Lego as well as many other ACME Clients.

That allows you to get your cert but it still leaves an unknown routing problem that might affect other people who wish to connect to you.

I am a volunteer so do not have access to LE's servers. And, I can't recall a time where LE staff helped debug a timeout failure to a single location. Perhaps they will for this case I am just setting your expectations. Maybe we will see other failures to Norway which would help isolate the problem.

We can reasonably say that the network problem is closer to you than to LE's server. Why? Because if the problem was near LE there would be many failures and those would trigger LE's internal alerting system. We haven't seen anyone else report problems yet and we don't see any mention of a problem on the LE status board.

Since you can readily reproduce the problem perhaps asking your ISP to check a successful test from LE staging vs production. They know the destination (you) and can check traffic further upstream from you. These problems can be very difficult to debug even for experienced comms support.

UPDATE:
Just checking cert history I see about 200 certs issued by Let's Encrypt in the past 30 days for subdomains of nscale.com. Even some issued today and in recent days. If you are the operator of nscale do you have any insights to what is different about this domain/cert request than others?

3

(post deleted by author)