Unblock IP address

It's not in our big blocklist. I'll check the CDN shortly, but i think that list is kept clear now.

5 Likes

It's not in either of our blocklists.

4 Likes

Hmm. Let's try this traceroute. Your earlier example was very short indicating a local problem. Even from my test US server the below traceroute is 13 steps. What does this traceroute show for you?

sudo traceroute -T -p443 acme-v02.api.letsencrypt.org

You also noted you can reach that endpoint from another IP. Are there any other differences in your local network between the working and failing IP?

3 Likes

And this is what I see.

sudo traceroute -T -p443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  EdgeRouter-4 (192.168.1.1)  0.238 ms  0.279 ms  0.216 ms
 2  96.120.60.137 (96.120.60.137)  13.805 ms  13.799 ms  13.791 ms
 3  162.151.125.157 (162.151.125.157)  11.590 ms  13.775 ms  13.767 ms
 4  ae-2-rur02.beaverton.or.bverton.comcast.net (68.85.243.154)  15.744 ms  15.735 ms  15.727 ms
 5  96.216.60.245 (96.216.60.245)  13.713 ms  13.645 ms  13.697 ms
 6  68.85.243.197 (68.85.243.197)  15.016 ms  17.661 ms  17.650 ms
 7  69.252.236.134 (69.252.236.134)  14.535 ms  9.776 ms  9.945 ms
 8  172.65.32.248 (172.65.32.248)  9.187 ms  9.922 ms  9.171 ms
2 Likes

I can't help wondering if a Palo Alto Networks firewall is involved.

We have had problems with that brand blocking inbound requests to servers as they introduced a new feature in their Application rules for "acme protocol" earlier this year. Different models of their firewall block in slightly different ways.

I don't have any trouble simulating acme challenge requests to your server. But, maybe Palo Alto Networks, or another firewall, is blocking outbound challenge domain names?

This could explain your short traceroute and your ability to reach google from that IP.

3 Likes

Unfortunately, LetsEncrypt does not have a http service running on the api endpoints. IMHO, it would be great for LetsEncrypt/Boulder expose at least a "hello.txt" on port 80, so connectivity issues could be resolved with something like

curl http://acme-v02.api.letsencrypt.org/hello.txt

Note the http, not https in the url.

Under the current deployment, curl http://acme-v02.api.letsencrypt.org/ from a usable connection will return one of two random respones:

curl: (56) Recv failure: Connection reset by peer

or

curl: (52) Empty reply from server

I don't know what a firewall blocked connection would show up as, and it might be a (56).

I still believe this is due to out-of-date software libraries. Errors such as NSS error -5978 (PR_NOT_CONNECTED_ERROR) and curl: (35) Network file descriptor is not connected are very common to outdated software, and often resolved by an upgrade to NSS/openssl/libcurl/etc.

It could be. I just realized there is no exception shared from the ACME client.

Edit: I'm proposing a Feature for ISRG to host a file on port 80 behind their firewall. see Request: Serve a debug text file via http behind ISRG's firewall. Removing HTTPS from the equation would greatly simplify debugging a lot of dropped connection and "unblock!" issues.

4 Likes

I am aware. But, how does that relate to tests shown in this thread? I don't see anyone that tried that.

It might be old buggy software. But, remember that both curl and openssl failed to connect. Is there some specific component they should check? The openssl version is older but it doesn't look any older than the one I have on my RHEL derivative.

And, yeah, the failure from the ACME client would be interesting. We hopped over some steps (the acme messages) we normally see which might give further clues.

@CGC On the IP that worked, is it using the same versions of curl, openssl and the like?

3 Likes

No one did. I'm just trying to illustrate that could easily have differentiated a connection issue.

Copy/Paste the error codes that @CGC shared into a search engine. There is a rich global history of those errors being from outdated software, and being resolved when software is updated.

I'm not familiar with their Linux Distro. In my experience, distro maintainers do a bit of patching and reworking of files and frameworks. While the openssl result suggests it could be a firewall issue like you bought up, the extremely old libcurl version and rather old openssl version suggest there still may be some library issue.

It's not even the ACME messages - if they're using Certbot, the logged exceptions tend to give a lot of info. The requests library tends to error a certain way during a Palo Alto firewall issue vs an ISRG firewall issue.

3 Likes

Agreed. Often "Unexpected EOF" error due to the way the ISRG firewall blocks.

But, LE staff has already said the IP is not blocked. They have only very rarely been wrong about that (only once that I recall when it involved a range of IP's blocked).

4 Likes

101.133.166.108

Welcome @sujiany but please create a new Help topic for your problem. Answer the questions on the form you will be shown as best you can.

4 Likes

Hi All, the issue has been resolved, as per your emails the issue was our edge firewall. We also moved to another more up-to-date server.

Thank you all :slight_smile:

2 Likes