Can't reach letsencrypt from a single host

staylor · August 21, 2022, 5:06pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
control-1.apa.moejo.io

I ran this command:
certbot --apache

It produced this output:
root@control-1:~# certbot --apache

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
An unexpected error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 314, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b40ed83a0>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b40ed83a0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1088, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b40ed83a0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

ValueError: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version):
root@control-1:~# apache2 -v
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2022-06-14T13:30:55

The operating system my web server runs on is (include version):
root@control-1:~# lsb_release -a
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

My hosting provider, if applicable, is:
Digital Ocean

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
root@control-1:~# certbot --version
certbot 0.40.0

Additional information:

From another host that I run, also on digital ocean with same setup, no problems:

root@east-1:~# mtr -n -r -4 -c 50 acme-v02.api.letsencrypt.org
Start: 2022-08-21T15:44:16+0000
HOST: east-1                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                       100.0    50    0.0   0.0   0.0   0.0   0.0
  2.|-- 10.71.4.86                 0.0%    50    0.5   0.9   0.3  18.0   2.5
  3.|-- 138.197.248.84             0.0%    50    0.8   2.0   0.7  18.0   3.5
  4.|-- 138.197.251.116            0.0%    50    0.6   1.3   0.4  35.7   5.0
  5.|-- 138.197.244.15             0.0%    50    7.2   1.6   1.0   8.9   1.4
  6.|-- 192.241.164.73             0.0%    50    8.1   3.8   1.4  18.7   4.2
  7.|-- 172.70.112.4               0.0%    50    2.0   3.4   1.1  28.2   5.2
  8.|-- 172.65.32.248              0.0%    50    1.1   1.1   1.0   1.5   0.1

From This (Affected) host:

root@control-1:~# mtr -n -r -4 -c 50 acme-v02.api.letsencrypt.org
Start: 2022-08-21T10:44:30-0500
HOST: control-1                   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                       100.0    50    0.0   0.0   0.0   0.0   0.0
  2.|-- 10.70.4.48                 0.0%    50    0.5   3.4   0.4  66.8  11.0
  3.|-- 138.197.251.70             0.0%    50    1.0   1.6   0.9  21.1   3.0
  4.|-- 138.197.251.110            0.0%    50    1.0   1.0   0.5  12.3   1.7
  5.|-- ???                       100.0    50    0.0   0.0   0.0   0.0   0.0

root@control-1:~# ufw status verbose
Status: inactive

root@control-1:~# iptables -nvL --line-numbers
Chain INPUT (policy ACCEPT 145 packets, 13172 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1    22248 2818K f2b-sshd   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 129 packets, 16824 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain f2b-sshd (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1       22  1728 REJECT     all  --  *      *       51.83.99.204         0.0.0.0/0            reject-with icmp-port-unreachable
2       25  1952 REJECT     all  --  *      *       45.148.122.228       0.0.0.0/0            reject-with icmp-port-unreachable
3       27  1980 REJECT     all  --  *      *       43.135.8.135         0.0.0.0/0            reject-with icmp-port-unreachable
4       27  2072 REJECT     all  --  *      *       126.77.170.137       0.0.0.0/0            reject-with icmp-port-unreachable
5       26  1964 REJECT     all  --  *      *       43.154.99.157        0.0.0.0/0            reject-with icmp-port-unreachable
6       28  2040 REJECT     all  --  *      *       157.230.245.64       0.0.0.0/0            reject-with icmp-port-unreachable
7        9   568 REJECT     all  --  *      *       104.175.239.228      0.0.0.0/0            reject-with icmp-port-unreachable
8    14533 2181K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

root@control-1:~# netstat -r

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         192.34.60.1     0.0.0.0         UG        0 0          0 eth0
10.10.0.0       0.0.0.0         255.255.0.0     U         0 0          0 eth0
10.136.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth1
192.34.60.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

This started happening last week.
It appears to be intermittent. Mostly borked, but intermittently I can reach the renewal server for a few seconds.
I am not aware of any changes that may have affected our side.

Thank you for any guidance.

My initial hypothesis is that this is happening outside of my network .. see hop 5 on the MTR?

Scotty Taylor

rg305 · August 22, 2022, 1:24am

Hi @staylor, and welcome to the LE community forum

Based on the information provided, I agree with your hypothesis.
There shouldn't be any such loss after line #2.

You might want to try a test that can specifically check the MTU size allowed.

staylor · August 22, 2022, 3:27pm

Update:

I escalated this with the affected network operator (Digital Ocean -- their backbone network appeared to be the issue).

They denied any problems but it suddenly works properly now.

The issue appears to have been in a downline network hop and not related to my host, my letsencrypt setup, or letsencrypt's setup or network and has been resolved.

Thanks to all.

St

system · September 21, 2022, 3:27pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to get a cert Help	10	287	June 14, 2024
Unable to run sudo certbot --apache on Ubuntu 18.04 Help	3	1722	May 24, 2019
Challenge failed for domain Error Help	5	2068	May 30, 2020
Lets Encrypt Certbot cannot create new certs Help	6	3653	November 27, 2018
404 Error LetsEncrypt Help	7	1302	June 2, 2020

Can't reach letsencrypt from a single host

Related topics