It's not in our big blocklist. I'll check the CDN shortly, but i think that list is kept clear now.
It's not in either of our blocklists.
Hmm. Let's try this traceroute. Your earlier example was very short indicating a local problem. Even from my test US server the below traceroute is 13 steps. What does this traceroute show for you?
sudo traceroute -T -p443 acme-v02.api.letsencrypt.org
You also noted you can reach that endpoint from another IP. Are there any other differences in your local network between the working and failing IP?
And this is what I see.
sudo traceroute -T -p443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 EdgeRouter-4 (192.168.1.1) 0.238 ms 0.279 ms 0.216 ms
2 96.120.60.137 (96.120.60.137) 13.805 ms 13.799 ms 13.791 ms
3 162.151.125.157 (162.151.125.157) 11.590 ms 13.775 ms 13.767 ms
4 ae-2-rur02.beaverton.or.bverton.comcast.net (68.85.243.154) 15.744 ms 15.735 ms 15.727 ms
5 96.216.60.245 (96.216.60.245) 13.713 ms 13.645 ms 13.697 ms
6 68.85.243.197 (68.85.243.197) 15.016 ms 17.661 ms 17.650 ms
7 69.252.236.134 (69.252.236.134) 14.535 ms 9.776 ms 9.945 ms
8 172.65.32.248 (172.65.32.248) 9.187 ms 9.922 ms 9.171 ms
I can't help wondering if a Palo Alto Networks firewall is involved.
We have had problems with that brand blocking inbound requests to servers as they introduced a new feature in their Application rules for "acme protocol" earlier this year. Different models of their firewall block in slightly different ways.
I don't have any trouble simulating acme challenge requests to your server. But, maybe Palo Alto Networks, or another firewall, is blocking outbound challenge domain names?
This could explain your short traceroute and your ability to reach google from that IP.
Unfortunately, LetsEncrypt does not have a http service running on the api endpoints. IMHO, it would be great for LetsEncrypt/Boulder expose at least a "hello.txt" on port 80, so connectivity issues could be resolved with something like
curl http://acme-v02.api.letsencrypt.org/hello.txt
Note the http, not https in the url.
Under the current deployment, curl http://acme-v02.api.letsencrypt.org/
from a usable connection will return one of two random respones:
curl: (56) Recv failure: Connection reset by peer
or
curl: (52) Empty reply from server
I don't know what a firewall blocked connection would show up as, and it might be a (56).
I still believe this is due to out-of-date software libraries. Errors such as NSS error -5978 (PR_NOT_CONNECTED_ERROR)
and curl: (35) Network file descriptor is not connected
are very common to outdated software, and often resolved by an upgrade to NSS/openssl/libcurl/etc.
It could be. I just realized there is no exception shared from the ACME client.
Edit: I'm proposing a Feature for ISRG to host a file on port 80 behind their firewall. see Request: Serve a debug text file via http behind ISRG's firewall. Removing HTTPS from the equation would greatly simplify debugging a lot of dropped connection and "unblock!" issues.
I am aware. But, how does that relate to tests shown in this thread? I don't see anyone that tried that.
It might be old buggy software. But, remember that both curl and openssl failed to connect. Is there some specific component they should check? The openssl version is older but it doesn't look any older than the one I have on my RHEL derivative.
And, yeah, the failure from the ACME client would be interesting. We hopped over some steps (the acme messages) we normally see which might give further clues.
@CGC On the IP that worked, is it using the same versions of curl, openssl and the like?
No one did. I'm just trying to illustrate that could easily have differentiated a connection issue.
Copy/Paste the error codes that @CGC shared into a search engine. There is a rich global history of those errors being from outdated software, and being resolved when software is updated.
I'm not familiar with their Linux Distro. In my experience, distro maintainers do a bit of patching and reworking of files and frameworks. While the openssl result suggests it could be a firewall issue like you bought up, the extremely old libcurl version and rather old openssl version suggest there still may be some library issue.
It's not even the ACME messages - if they're using Certbot, the logged exceptions tend to give a lot of info. The requests
library tends to error a certain way during a Palo Alto firewall issue vs an ISRG firewall issue.
Agreed. Often "Unexpected EOF" error due to the way the ISRG firewall blocks.
But, LE staff has already said the IP is not blocked. They have only very rarely been wrong about that (only once that I recall when it involved a range of IP's blocked).
101.133.166.108
Welcome @sujiany but please create a new Help topic for your problem. Answer the questions on the form you will be shown as best you can.
Hi All, the issue has been resolved, as per your emails the issue was our edge firewall. We also moved to another more up-to-date server.
Thank you all