Suddenly "Network is unreachable"

[Blake1]

Been working fine on this server, now it’s not connecting?

It produced this output:
An unexpected error occurred:
ConnectionError: HTTPSConnectionPool(host=‘acme-v01.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError(’<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb069529b50>: Failed to establish a new connection: [Errno 101] Network is unreachable’,))
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version):
Apache/2.4.18 (Ubuntu)

The operating system my web server runs on is (include version):
Xenial

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes

curl https://acme-v01.api.letsencrypt.org/directory
curl: (7) Failed to connect to acme-v01.api.letsencrypt.org port 443: Connection refused

From the log files:
2017-08-20 21:59:18,773:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.configurator.ApacheConfigurator object at 0x7fb06c94a810> and installer <certbot_apache.configurator.ApacheConfigurator object at 0x7fb06c94a810>
2017-08-20 21:59:18,779:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, contact=(u’mailto:btcarver@lisnews.com’,), agreement=u’https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf’, key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7fb069597dd0>)>)), uri=u’https://acme-v01.api.letsencrypt.org/acme/reg/8299485’, new_authzr_uri=u’https://acme-v01.api.letsencrypt.org/acme/new-authz’, terms_of_service=u’https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf’), 9471dcdb61fb97f16adb2bd294711dbf, Meta(creation_host=u’ ‘, creation_dt=datetime.datetime(2017, 1, 13, 3, 4, 42, tzinfo=)))>
2017-08-20 21:59:18,781:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2017-08-20 21:59:18,788:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-08-20 21:59:24,818:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/root/.local/share/letsencrypt/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 753, in main
return config.func(config, plugins)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 598, in run
le_client = _init_le_client(config, authenticator, installer)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 397, in _init_le_client
return client.Client(config, acc, authenticator, installer, acme=acme)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/client.py”, line 235, in init
acme = acme_from_config_key(config, self.account.key)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/client.py”, line 45, in acme_from_config_key
return acme_client.Client(config.server, key=key, net=net)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 71, in init
self.net.get(directory).json())
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 654, in get
self._send_request(‘GET’, url, **kwargs), content_type=content_type)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 627, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/requests/sessions.py”, line 488, in request
resp = self.send(prep, **send_kwargs)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/requests/sessions.py”, line 609, in send
r = adapter.send(request, **kwargs)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/requests/adapters.py”, line 487, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host=‘acme-v01.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError(’<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb069529b50>: Failed to establish a new connection: [Errno 101] Network is unreachable’,))

[schoen]

Hi @Blake1,

This implies that your server can’t connect to Let’s Encrypt, rather than the other way around, as you also confirmed with your curl command.

Can you use curl to connect to other HTTPS web sites? Is it different if you use curl -4 or curl -6?

[Blake1]

I can indeed curl some other random sites, and - 4 or -6 don’t seem to change much when I try
curl -6 https://acme-v01.api.letsencrypt.org/directory
curl: (7) Couldn’t connect to server
curl https://acme-v01.api.letsencrypt.org/directory
curl: (7) Failed to connect to acme-v01.api.letsencrypt.org port 443: Connection refused

It seems to just fail on acme-v01.api.letsencrypt.org

[rg305]

Try something simpler:
nslookup acme-v01.api.letsencrypt.org

If that fails, there is the reason this fails.
If that is successful, try:
curl https://ipv4.address
curl -6 https://ipv6.address

For IPv4 I get:
curl https://23.219.99.47
curl: (51) SSL: certificate subject name (*.api.letsencrypt.org) does not match target host name ‘23.219.99.47’

[Blake1]

nslookup succeeds.
Both curl for ipv4 and ipv6 fail with "could not resolve host"
Just tried curl to google yahoo and wikipedia and all three succeeded.

[Blake1]

Oh darn it all, sorry, turns out it to be something in the firewall. Not sure what changed, but something because I just tried disabling it and POOF like magic certbot is working again.

Sorry to waste your time on this one.

[Blake1]

Other than 443, what ports do I need open?

[Patches]

Certbot communicates outbound only on port 443.

Inbound it needs one of port 443 for tls-sni-01/apache/nginx/standalone verification, port 80 for http-01/webroot verification, or port 53 (on your DNS server) for dns-01 verification.

[Blake1]

Found the trouble, somehow had 23.231.0.0/8 blocked in the firewall!
Thank you again for helping everyone.

[rg305]

Wow!
That means 23.*.*.* was being blocked.

Now the question is… who adds /8 entries as blocked in a firewall?
But that’s for a different thread altogether.

Glad to see you got it sorted out