Unblock IP address

grep -Ei 'ISRG|DST|R3' /etc/pki/tls/certs/ca-bundle.crt | grep -e '#'
# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1
curl -vv -I https://acme-v02.api.letsencrypt.org/directory
* About to connect() to acme-v02.api.letsencrypt.org port 443 (#0)
*   Trying 172.65.32.248...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5978 (PR_NOT_CONNECTED_ERROR)
* Network file descriptor is not connected
* Closing connection 0
curl: (35) Network file descriptor is not connected
2 Likes

I think it's possible that your very old version of curl can not properly handle the TLS connection. You also have a very old OpenSSL. You're on the old 1.02 branch and patch k - they ultimately got up to patch v. IMHO, it's odd that it's running a fips version, but not the latest fips version.

In any event, there may be issues with your machine's ability to handle protocols and ciphers. That would explain why you can access some https sites but not others.

Looking at the curl changelog - there were a lot of improvements to tls1.1 and 1.2 in subsequent releases.

You could try the following, but I don't think your version of curl may support these flags:

curl --tlsv1.1 -tls-max 1.1 https://acme-staging-v02.api.letsencrypt.org/directory

curl --tlsv1.2 -tls-max 1.2 https://acme-staging-v02.api.letsencrypt.org/directory

I forgot to mention:

The reason why everyone here is focused on curl, is because what you shared earlier does not look like what happens when an ip address is blocked

3 Likes

Let's try connecting with something other than curl. Can you show result of this?

 echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head -10

EDIT:
I'm starting to think your IP may be blocked. Curious to see openssl result still. Is this the machine you have been using to regularly renew the certs for mercurycolleges.nsw.edu.au as seen here ?

4 Likes
echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head -10
write:errno=104
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported


curl --tlsv1.1 -tls-max 1.1 https://acme-staging-v02.api.letsencrypt.org/directory
error code: 1003curl: (35) Network file descriptor is not connected


curl --tlsv1.2 -tls-max 1.2 https://acme-staging-v02.api.letsencrypt.org/directory
curl: (56) Recv failure: Connection reset by peer
curl: (35) Network file descriptor is not connected

Is the IP of your requesting machine as in DNS for mercurycolleges.nsw.edu.au

EDIT: @cgc I think it's likely you are blocked but please confirm the IP address to check. Thanks

3 Likes

Hi Mike, The IP is 91.239.243.28 thank you

2 Likes

@lestaff Will you please check if this IP is blocked.

3 Likes

It's not in our big blocklist. I'll check the CDN shortly, but i think that list is kept clear now.

5 Likes

It's not in either of our blocklists.

4 Likes

Hmm. Let's try this traceroute. Your earlier example was very short indicating a local problem. Even from my test US server the below traceroute is 13 steps. What does this traceroute show for you?

sudo traceroute -T -p443 acme-v02.api.letsencrypt.org

You also noted you can reach that endpoint from another IP. Are there any other differences in your local network between the working and failing IP?

3 Likes

And this is what I see.

sudo traceroute -T -p443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  EdgeRouter-4 (192.168.1.1)  0.238 ms  0.279 ms  0.216 ms
 2  96.120.60.137 (96.120.60.137)  13.805 ms  13.799 ms  13.791 ms
 3  162.151.125.157 (162.151.125.157)  11.590 ms  13.775 ms  13.767 ms
 4  ae-2-rur02.beaverton.or.bverton.comcast.net (68.85.243.154)  15.744 ms  15.735 ms  15.727 ms
 5  96.216.60.245 (96.216.60.245)  13.713 ms  13.645 ms  13.697 ms
 6  68.85.243.197 (68.85.243.197)  15.016 ms  17.661 ms  17.650 ms
 7  69.252.236.134 (69.252.236.134)  14.535 ms  9.776 ms  9.945 ms
 8  172.65.32.248 (172.65.32.248)  9.187 ms  9.922 ms  9.171 ms
2 Likes

I can't help wondering if a Palo Alto Networks firewall is involved.

We have had problems with that brand blocking inbound requests to servers as they introduced a new feature in their Application rules for "acme protocol" earlier this year. Different models of their firewall block in slightly different ways.

I don't have any trouble simulating acme challenge requests to your server. But, maybe Palo Alto Networks, or another firewall, is blocking outbound challenge domain names?

This could explain your short traceroute and your ability to reach google from that IP.

3 Likes

Unfortunately, LetsEncrypt does not have a http service running on the api endpoints. IMHO, it would be great for LetsEncrypt/Boulder expose at least a "hello.txt" on port 80, so connectivity issues could be resolved with something like

curl http://acme-v02.api.letsencrypt.org/hello.txt

Note the http, not https in the url.

Under the current deployment, curl http://acme-v02.api.letsencrypt.org/ from a usable connection will return one of two random respones:

curl: (56) Recv failure: Connection reset by peer

or

curl: (52) Empty reply from server

I don't know what a firewall blocked connection would show up as, and it might be a (56).

I still believe this is due to out-of-date software libraries. Errors such as NSS error -5978 (PR_NOT_CONNECTED_ERROR) and curl: (35) Network file descriptor is not connected are very common to outdated software, and often resolved by an upgrade to NSS/openssl/libcurl/etc.

It could be. I just realized there is no exception shared from the ACME client.

Edit: I'm proposing a Feature for ISRG to host a file on port 80 behind their firewall. see Request: Serve a debug text file via http behind ISRG's firewall. Removing HTTPS from the equation would greatly simplify debugging a lot of dropped connection and "unblock!" issues.

4 Likes

I am aware. But, how does that relate to tests shown in this thread? I don't see anyone that tried that.

It might be old buggy software. But, remember that both curl and openssl failed to connect. Is there some specific component they should check? The openssl version is older but it doesn't look any older than the one I have on my RHEL derivative.

And, yeah, the failure from the ACME client would be interesting. We hopped over some steps (the acme messages) we normally see which might give further clues.

@CGC On the IP that worked, is it using the same versions of curl, openssl and the like?

3 Likes

No one did. I'm just trying to illustrate that could easily have differentiated a connection issue.

Copy/Paste the error codes that @CGC shared into a search engine. There is a rich global history of those errors being from outdated software, and being resolved when software is updated.

I'm not familiar with their Linux Distro. In my experience, distro maintainers do a bit of patching and reworking of files and frameworks. While the openssl result suggests it could be a firewall issue like you bought up, the extremely old libcurl version and rather old openssl version suggest there still may be some library issue.

It's not even the ACME messages - if they're using Certbot, the logged exceptions tend to give a lot of info. The requests library tends to error a certain way during a Palo Alto firewall issue vs an ISRG firewall issue.

3 Likes

Agreed. Often "Unexpected EOF" error due to the way the ISRG firewall blocks.

But, LE staff has already said the IP is not blocked. They have only very rarely been wrong about that (only once that I recall when it involved a range of IP's blocked).

4 Likes

101.133.166.108

Welcome @sujiany but please create a new Help topic for your problem. Answer the questions on the form you will be shown as best you can.

4 Likes

Hi All, the issue has been resolved, as per your emails the issue was our edge firewall. We also moved to another more up-to-date server.

Thank you all :slight_smile:

2 Likes