10:48:53-10:49:11, token HQrQFZ0NONJk0-sltgaq9hM2KNawm50htMsj059Tbis
My web server is: Apache, serving the webroot directly (nothing unusual in vhost/DocumentRoot config - confirmed by staging succeeding with the identical setup seconds earlier).
The operating system my web server runs on is: Ubuntu 22.04 LTS (containerized ACME client via Docker).
My hosting provider, if applicable, is: self-hosted, private data center (OpenStack compute node), not a public cloud.
I can login to a root shell on my machine: yes.
I'm using a control panel to manage my site: no.
The version of my client is: lego (containerized).
Additional context, since this doesn't match a typical firewall-block pattern and we'd like to understand what's happening on your side specifically:
The error carries no "During secondary validation:" prefix, so it appears to be the Primary validator itself timing out, not a secondary perspective.
193.143.78.6 is reachable from 25/25 diverse external check-host.net vantage points on TCP/80, and correctly serves a 404 for a bogus challenge token from 12/12 HTTP probes. It is not a general reachability problem.
During the successful staging run seconds earlier, our perimeter firewall logged all 5 validator connections cleanly (allowed, clean TCP close): source IPs 13.213.78.76 (Singapore), 66.133.109.36, 3.148.226.41, 13.63.62.195 (Sweden), 54.201.8.134 (all US except noted). This is the reference signature for genuine validator traffic reaching us successfully.
During both production failures, our firewall logs show zero connections from any of those IPs, or any other source, to port 80 on this address, in the exact failure windows. This isn't a case of "arrived and got blocked" - nothing arrives at all.
We've directly ruled out, on our own infrastructure: geoblocking (no source-region restriction on the allow rule, no relevant deny rules above it), flood/connection-rate protection (thresholds orders of magnitude above this traffic volume), packet buffer protection (enabled, but zero related events logged in either failure window), and asymmetric-path session handling (deliberately disabled for a direct test - failure persisted identically).
A live, fully-instrumented paired test (firewall logs, packet capture at our edge and at our host, tcpdump on the host) is planned to catch this happening again, but as of today it has not happened during an already-completed capture window - the failures happen in a matter of seconds and our capture point had to be relocated mid-test on one occasion.
Given the above, we're trying to understand whether this could be an issue on Let's Encrypt's Primary validator's own outbound path (e.g. a peering/interconnect issue with the network(s) upstream of us), similar to a pattern described in an older community thread (Timeout during connect (likely firewall problem)) where a carrier-level interconnect problem was suspected. If you're able to look up either of the two failed authorizations above and tell us what your validator observed on its end (e.g. no response to SYN, a response that never completed, or something else), that would help us rule our own infrastructure in or out conclusively. Happy to provide any further detail.
Welcome @ssadiq Thanks for a terrific report. I agree that the most likely problem is some network routing problem between the LE primary center and your site.
You linked to a thread where I'd commented on a similar problem. And, I don't have a lot more to add than what I said there. But, I do have some ...
First, are you a customer of nscale or are you part of their support team?
Second, I see you are using Cloudflare as your DNS provider. You could switch to using a DNS Challenge instead. That avoids LE having to send HTTP requests for validation. Cloudflare is well-supported by Lego as well as many other ACME Clients.
That allows you to get your cert but it still leaves an unknown routing problem that might affect other people who wish to connect to you.
I am a volunteer so do not have access to LE's servers. And, I can't recall a time where LE staff helped debug a timeout failure to a single location. Perhaps they will for this case I am just setting your expectations. Maybe we will see other failures to Norway which would help isolate the problem.
We can reasonably say that the network problem is closer to you than to LE's server. Why? Because if the problem was near LE there would be many failures and those would trigger LE's internal alerting system. We haven't seen anyone else report problems yet and we don't see any mention of a problem on the LE status board.
Since you can readily reproduce the problem perhaps asking your ISP to check a successful test from LE staging vs production. They know the destination (you) and can check traffic further upstream from you. These problems can be very difficult to debug even for experienced comms support.
UPDATE:
Just checking cert history I see about 200 certs issued by Let's Encrypt in the past 30 days for subdomains of nscale.com. Even some issued today and in recent days. If you are the operator of nscale do you have any insights to what is different about this domain/cert request than others?
I am seeing the same production-only timeout pattern: staging validates immediately, while the production run reports a connect timeout despite the identical webroot. The repeated failure after a successful staging validation makes this look like a validator-path or timing issue rather than a straightforward local firewall block.
On the DNS point: nscale.com's zone is hosted on Cloudflare for DNS management, but this particular subdomain's A record points straight at our own infrastructure, it isn't proxied. That said, you're right that it doesn't explain or fix the underlying routing problem, so we're continuing to chase that in parallel rather than treat DNS-01 as the end of it.
Agreed on the "closer to us than to LE" reasoning, that matches what we've found locally. We've been through our own firewall and network config in detail this week: no geo or EDL rule matches this, flood/DoS thresholds are nowhere near the traffic involved, and a live packet capture during an actual failed validation showed nothing arriving at our firewall or switches at all. So we can't find a local block, which points the same direction you're describing.
We haven't raised with our transit provider as it is hard to pinpoint where the issue lies. It may not be our immediate provider but an upstream ISP.
On the 200 certs question: as far as we've found, the difference isn't the account or the ACME setup, it's the physical site. This domain sits at a different data centre to the others, which all appear to be renewing on schedule. That's the one thing that stands out so far.
The affected host is test.compute.sta1.nscale.com, as shown in my original report. It began on 13 Aug and still reproduces only against production, with staging succeeding from the same webroot.
So @Jordan.Brooks3 are you also @ssadiq ? Or at least on the same tech team? Because you weren't the first poster in this thread and did not provide a hostname in your first post.
Your first post in this thread said "I am seeing the same ...". I thought that meant you were posting about a different site with the same symptoms as described by @ssadiq.
That said, @ssadiq has provided very good info and sounds like they've isolated it to something about a specific local center.
@MikeMcQ unfortunately I still haven't been able to resolve the problem. I was hoping someone with LE inside knowledge may be able to point me in the right direction or preferably a workaround.
I mentioned using a DNS Challenge so that's one option. You do not need to proxy your domain at Cloudflare to use that. Be sure to evaluate the security risk by needing to store the API token for the ACME Client. A DNS Challenge is not dependent on HTTP traffic so can be used to get a cert at any location and deploy (copy) it elsewhere. A DNS Challenge can also obtain a wildcard cert which may simplify admin.
Another option is to use a different Certificate Authority. Google Trust Services is free and reliable. ZeroSSL is another free CA people use with ACME. I personally use GTS along with LE regularly.
lego --email=notifications@example.com --domains=test.compute.sta1.nscale.com --server https://acme-staging-v02.api.letsencrypt.org/directory --http.webroot /etc/letsencrypt/http-01 --accept-tos --http run