Possible IP blocklist?

Help
1

Hello,

I contact you in regard of an issue we're recently facing on one of our servers. We've been using let's encrypt on that server, as well as others for years and never got any issues. However since about a week we're unable to contact letsencrypt servers to validate new subdomains or renew.

It all started when we tried to deploy a new application under a new subdomain name running "certbot --apache" and receive a timeout error. We've then made some test of our configuration but we have the same OS version, certbot version, etc... on other server and they're all working just fine.

After looking at some post on this forum we think that we might have been blacklisted as we transferred a domain name from that server to a new one without unregistering the ssl renewal on the previous; thus making validation requests on the wrong domain name / IP. It has been removed yesterday and now we don't have any queries for invalid domain names.

Could you take a look and tell us if our current issue is related to that or, if not, if you have any idea about what is going on?

Here's the ip of our server: 118.31.173.12

2

We're not blocking this IP address.

4 Likes
3

Thanks @JamesLE for the quick answer. Do you have any suggestion about what could be the issue there ? Yesterday, most of the time "certbot --apache" would just timeout at once, but today it starts to list the domain names with the "Which names would you like to activate HTTPS for?" message and only after the selection it timeout.

I've manually run some curl ("curl https://acme-v02.api.letsencrypt.org/directory") and it seems to only respond from time to time. Have you heard of any other strange network issues like that recently ?

Here is some more context:
Server provider: Aliyun
Server location: China
OS: Ubuntu 20.04.4 LTS
certbot version: certbot 1.29.0
IP: 118.31.173.12
Error message: An unexpected error occurred: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

1 Like
4

One potential reason:
Minor change to validation networking - API Announcements - Let's Encrypt Community Support (letsencrypt.org)

2 Likes
5

@rg305 That crossed my mind initially too, but I believe that change is just for the validation servers doing the actual validation. I.e.: outgoing connections from LEs point of view. And not for the API frontend, which is already behind Cloudflare to begin with. (I.e.: incoming connections from LEs point of view.)

4 Likes
6

FYI - @JamesLE alternative for blacklist to be more PC; I am not judging you or anyone just sharing information from projects I've been on that were required to alternative words.
Why changing the terms blacklist and whitelist isn't as easy as it might seem.
and
SAP sets alternatives to master/slave, blacklist/whitelist

3 Likes
7

And The Linux Foundation has taken action too.

4 Likes
8

Bruce is correct, we don't use the term "blacklist", we say "blocklist". The title of this thread was created by the original poster AntoineG, not by LE staff, but I'll edit it anyway.

8 Likes
9

Thanks for the rename of the threat and the clarification on naming convention. Do you have any idea about what could be the issue that we're facing ? Or at least any advice on what we could we do to troubleshoot / identify the main issue ?

10

Please show us the complete error message.

2 Likes
11

Here's what I'm receiving right now:

$ certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

12

The full details in the logs is:

2022-08-03 10:24:11,743:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connection.py", line 416, in connect
    self.sock = ssl_wrap_socket(
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/snap/certbot/2192/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/snap/certbot/2192/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/snap/certbot/2192/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:1114: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/certbot/2192/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/snap/certbot/2192/lib/python3.8/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/certbot/2192/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/2192/lib/python3.8/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/2192/lib/python3.8/site-packages/certbot/_internal/main.py", line 1744, in main
    return config.func(config, plugins)
  File "/snap/certbot/2192/lib/python3.8/site-packages/certbot/_internal/main.py", line 1434, in run
    le_client = _init_le_client(config, authenticator, installer)
  File "/snap/certbot/2192/lib/python3.8/site-packages/certbot/_internal/main.py", line 832, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
  File "/snap/certbot/2192/lib/python3.8/site-packages/certbot/_internal/client.py", line 311, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
  File "/snap/certbot/2192/lib/python3.8/site-packages/certbot/_internal/client.py", line 76, in acme_from_config_key
    client = acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/snap/certbot/2192/lib/python3.8/site-packages/acme/client.py", line 880, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/snap/certbot/2192/lib/python3.8/site-packages/acme/client.py", line 1242, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/snap/certbot/2192/lib/python3.8/site-packages/acme/client.py", line 1180, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/snap/certbot/2192/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/snap/certbot/2192/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/snap/certbot/2192/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)
2022-08-03 10:24:11,745:ERROR:certbot._internal.log:An unexpected error occurred:
2022-08-03 10:24:11,745:ERROR:certbot._internal.log:requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)
13

Do you have multiple versions of certbot installed?

What shows?:
curl https://acme-v02.api.letsencrypt.org/

Maybe we are dealing with a "great firewall" problem?:

2 Likes
14

It's usually on and off, here's the results of 2 curls launched one after the other with a few seconds interval.

$ curl https://acme-v02.api.letsencrypt.org/
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Boulder: The Let's Encrypt CA</title>

  <style type="text/css">
    header { display: flex; max-height: 30vh; flex-wrap: wrap; margin-bottom: 10vh; }
    header img { display: flex; max-height: 20vh; align-content: flex-end; margin-right: 20px; }
  </style>
</head>

<body>
  <header>
    <section>
      <img src="/static/images/LE-Logo-LockOnly.svg"/>
    </section>
    <section>
      <h1>Boulder<br>
      <small>The Let's Encrypt CA</small></h1>
    </section>
  </header>

  <section>
    <p>This is an <a href="https://tools.ietf.org/html/rfc8555">ACME</a> Certificate Authority running <a href="https://github.com/letsencrypt/boulder">Boulder</a>.</p>
    <p>This is a <em>programmatic</em> endpoint, an API for a computer to talk to. You should probably be using a specialized client to utilize the service, and not your web browser. See <a href="https://letsencrypt.org/docs"><tt>https://letsencrypt.org/docs</tt></a> for help.</p>
    <p>If you're trying to use this service, note that the starting point, <em>the directory</em>, is available at this URL: <a href="https://acme-v02.api.letsencrypt.org/directory"><tt>https://acme-v02.api.letsencrypt.org/directory</a></tt>.</p>
  </section>

  <footer>
      <p>
        <a href="https://letsencrypt.status.io" title="Status">Service Status (letsencrypt.status.io)</a> |
        <a href="https://twitter.com/letsencrypt" title="Twitter">Let's Encrypt Twitter</a>
      </p>
  </footer>

</body>
</html>




$ curl https://acme-v02.api.letsencrypt.org/
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to acme-v02.api.letsencrypt.org:443
15

What shows?:
ping -c 10 acme-v02.api.letsencrypt.org

2 Likes
16

Does your system have IPv4 and IPv6 ?

2 Likes
17

That one seems to always reply well

$ ping -c 10 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=49 time=153 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=49 time=154 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=49 time=148 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=49 time=156 ms
^C
--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3305ms
rtt min/avg/max/mdev = 148.018/152.738/156.150/2.972 ms
18

Yes, our system have both IPv4 and IPv6

19

Please let it run for all 10 tries.

Also, what shows?:
nslookup acme-v02.api.letsencrypt.org

2 Likes
20

That's quite a bit of hops [64-49=15 hops].

I get:
ttl=57
[64-57=7 hops (8 less hops)]

1 Like