ConnectTimeout: HTTPSConnectionPool (host='acme-v02.api.letsencrypt.org', port=443)

politas · September 11, 2018, 12:19pm

My domain is:

I ran this command:

certbot-auto

It produced this output:
Requesting to rerun /home/politas/bin/certbot-auto with root privileges…
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
An unexpected error occurred:
ConnectTimeout: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f3c091cda10>, ‘Connection to acme-v02.api.letsencrypt.org timed out. (connect timeout=45)’))
Please see the logfiles in /var/log/letsencrypt for more details.

cat /var/log/letsencrypt/letsencrypt.log
2018-09-11 22:12:05,262:DEBUG:certbot.main:certbot version: 0.27.1
2018-09-11 22:12:05,263:DEBUG:certbot.main:Arguments: []
2018-09-11 22:12:05,263:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-09-11 22:12:05,276:DEBUG:certbot.log:Root logging level set at 20
2018-09-11 22:12:05,277:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-09-11 22:12:05,277:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2018-09-11 22:12:05,363:DEBUG:certbot_apache.configurator:Apache version is 2.4.18
2018-09-11 22:12:05,694:DEBUG:certbot.plugins.disco:No installation (PluginEntryPoint#nginx):
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/plugins/disco.py”, line 132, in prepare
self._initialized.prepare()
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py”, line 139, in prepare
raise errors.NoInstallationError
NoInstallationError
2018-09-11 22:12:05,695:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f3c092717d0>
Prep: True
2018-09-11 22:12:05,696:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_debian.DebianConfigurator object at 0x7f3c092717d0> and installer <certbot_apache.override_debian.DebianConfigurator object at 0x7f3c092717d0>
2018-09-11 22:12:05,696:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2018-09-11 22:12:05,699:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=u’https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf’, only_return_existing=None, contact=(u’mailto:politas@gmail.com’,), key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f3c091fd5d0>)>)), uri=u’https://acme-v01.api.letsencrypt.org/acme/reg/3927102’, new_authzr_uri=u’https://acme-v01.api.letsencrypt.org/acme/new-authz’, terms_of_service=u’https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf’), a45bf6cd3464dc5669a8c66fa38ff4da, Meta(creation_host=u’fuller.lan’, creation_dt=datetime.datetime(2016, 9, 4, 5, 55, 7, tzinfo=)))>
2018-09-11 22:12:05,700:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2018-09-11 22:12:05,704:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2018-09-11 22:12:50,751:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1364, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1116, in run
le_client = _init_le_client(config, authenticator, installer)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 648, in _init_le_client
return client.Client(config, acc, authenticator, installer, acme=acme)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 247, in init
acme = acme_from_config_key(config, self.account.key, self.account.regr)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 50, in acme_from_config_key
return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/acme/client.py”, line 761, in init
directory = messages.Directory.from_json(net.get(server).json())
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/acme/client.py”, line 1095, in get
self._send_request(‘GET’, url, **kwargs), content_type=content_type)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/acme/client.py”, line 1044, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/requests/sessions.py”, line 488, in request
resp = self.send(prep, **send_kwargs)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/requests/sessions.py”, line 609, in send
r = adapter.send(request, **kwargs)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/requests/adapters.py”, line 479, in send
raise ConnectTimeout(e, request=request)
ConnectTimeout: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Max retries exceeded with url: /directory (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f3c091cda10>, ‘Connection to acme-v02.api.letsencrypt.org timed out. (connect timeout=45)’))
2018-09-11 22:12:50,752:ERROR:certbot.log:An unexpected error occurred:

My web server is (include version): Apache/2.4.18 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu Server 16.04 LTS

My hosting provider, if applicable, is: Self

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

politas · September 11, 2018, 12:24pm

I’ve been using two letsencrypt certs for years with no problems once I got the config sorted. Just recently, I’ve been unable to renew my certs.

JuergenAuer · September 11, 2018, 2:03pm

Hi @politas

I can't connect your site. Not http, not https.

But you have created a new certificate today:

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:flyingflux.net&lu=cert_search

11.09.2018.

cpu · September 11, 2018, 2:49pm

This looks like a connectivity error between your ACME client and the Let's Encrypt API. Can you verify that your server is able to successfully connect to other external sites?

Can you share the output from these commands (run on your server):

curl http://ipv4.whatismyip.akamai.com/ ; echo
curl http://ipv6.whatismyip.akamai.com/ ; echo
dig +short whoami.ipv4.akahelp.net TXT
dig +short whoami.ipv6.akahelp.net TXT
dig +short whoami.ds.akahelp.net TXT
dig +short whoami.ds.akahelp.net TXT
dig +short whoami.ds.akahelp.net TXT
mtr -c 20 -w -r acme-v02.api.letsencrypt.org

You may need to install a package to get the mtr command (I believe its mtr-tiny on Ubuntu/debian systems)

politas · September 12, 2018, 1:12am

I have ipv6 disabled on my network, so that will always fail:

$ curl http://ipv4.whatismyip.akamai.com/ ; echo
43.225.60.184

$ curl http://ipv6.whatismyip.akamai.com/ ; echo
curl: (7) Couldn't connect to server

$ dig +short whoami.ipv4.akahelp.net TXT
"ns" "172.68.116.82"

$ dig +short whoami.ipv6.akahelp.net TXT
"ns" "2400:cb00:26:1024::6ca2:f884"

$ dig +short whoami.ds.akahelp.net TXT
"ns" "2400:cb00:26:1024::6ca2:f85a"

$ mtr -c 20 -w -r acme-v02.api.letsencrypt.org
Start: Wed Sep 12 11:10:30 2018
HOST: fuller                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- edge                                                5.0%    20    0.6   0.6   0.4   0.8   0.0
  2.|-- lo0.lns01.470n.act.tsgnl.co                        90.0%    20   21.7  18.6  15.4  21.7   4.5
  3.|-- 202.137.170.2                                      90.0%    20   17.6  18.1  17.6  18.6   0.0
  4.|-- tengige0-0-0-0.civ-edge901.canberra.telstra.net    90.0%    20   22.5  21.9  21.2  22.5   0.0
  5.|-- bundle-ethernet2.dkn-edge901.canberra.telstra.net  90.0%    20   19.7  18.0  16.3  19.7   2.2
  6.|-- bundle-ethernet6.dkn-core1.canberra.telstra.net    90.0%    20   15.7  17.7  15.7  19.8   2.8
  7.|-- bundle-ether15.ken-core10.sydney.telstra.net       90.0%    20   28.1  28.9  28.1  29.6   1.0
  8.|-- bundle-ether1.ken-edge901.sydney.telstra.net       90.0%    20   24.5  24.9  24.5  25.2   0.0
  9.|-- 203.50.12.105                                       0.0%     2   29.1  28.3  27.5  29.1   1.0
 10.|-- 203.46.69.49                                        0.0%     2   50.7  65.6  50.7  80.5  21.0
 11.|-- a23-9-148-127.deploy.static.akamaitechnologies.com  0.0%     2   70.5  47.9  25.3  70.5  31.9

politas · September 12, 2018, 1:54am

Well, this is weird.
http://www.isitdownrightnow.com/flyingflux.net.html says it’s up.

http://www.downforeveryoneorjustme.com.au/index.php says it’s down.

I have port forwarding rules in place. Only thing I can think is that systems trying to use IPv6 are failing.

politas · September 12, 2018, 2:11am

Yes, the first part of the process, connecting to acme-v01.api.letsencrypt.org all seems to work fine. I'm not sure why the script uses acme-v02 later, but that's what seems to fail.

_az · September 12, 2018, 2:17am

Try run these from your server:

head -c 50 /dev/urandom | base64 | curl -v -d @- -X POST -i -m 10 -H 'Expect:' https://acme-v02.api.letsencrypt.org/acme/new-acct

vs

head -c 30000 /dev/urandom | base64 | curl -v -d @- -X POST -i -m 10 -H 'Expect:' https://acme-v02.api.letsencrypt.org/acme/new-acct

Let me know if the second one times out (it shouldn’t, they should both give HTTP 415).

Previously:

politas · September 12, 2018, 2:21am

They both return status 415

mnordhoff · September 12, 2018, 2:36am

acme-v01 and acme-v02 should be more or less exactly the same. Literally:

acme-v01.api.letsencrypt.org.        (insecure)  7200   CNAME  api.letsencrypt.org-ng.edgekey.net.
api.letsencrypt.org-ng.edgekey.net.  (insecure)  21600  CNAME  e14990.dscx.akamaiedge.net.
e14990.dscx.akamaiedge.net.          (insecure)  20     A      172.224.185.125
e14990.dscx.akamaiedge.net.          (insecure)  20     AAAA   2600:1402:16:38b::3a8e
e14990.dscx.akamaiedge.net.          (insecure)  20     AAAA   2600:1402:16:390::3a8e
e14990.dscx.akamaiedge.net.          (insecure)  20     AAAA   2600:1402:16:39c::3a8e

acme-v02.api.letsencrypt.org.        (insecure)  7200   CNAME  api.letsencrypt.org-ng.edgekey.net.
api.letsencrypt.org-ng.edgekey.net.  (insecure)  21600  CNAME  e14990.dscx.akamaiedge.net.
e14990.dscx.akamaiedge.net.          (insecure)  20     A      172.224.185.125
e14990.dscx.akamaiedge.net.          (insecure)  20     AAAA   2600:1402:16:38b::3a8e
e14990.dscx.akamaiedge.net.          (insecure)  20     AAAA   2600:1402:16:390::3a8e
e14990.dscx.akamaiedge.net.          (insecure)  20     AAAA   2600:1402:16:39c::3a8e

The vagaries of DNS resolution could point you at different endpoints, but it's unlikely to happen consistently.

What if you run "curl -v https://acme-v01.api.letsencrypt.org/directory" and "curl -v https://acme-v02.api.letsencrypt.org/directory"?

politas · September 12, 2018, 2:39am

$ curl -v https://acme-v01.api.letsencrypt.org/directory
*   Trying 23.9.148.127...
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
* Connected to acme-v01.api.letsencrypt.org (23.9.148.127) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 596 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: acme-v02.api.letsencrypt.org (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=acme-v02.api.letsencrypt.org
* 	 start date: Fri, 03 Aug 2018 01:36:30 GMT
* 	 expire date: Thu, 01 Nov 2018 01:36:30 GMT
* 	 issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /directory HTTP/1.1
> Host: acme-v01.api.letsencrypt.org
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 658
< Replay-Nonce: TJgriyhsr-ONxCNvhMgEdAjcsRH7q_zxtVuQR3szodU
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Wed, 12 Sep 2018 02:37:05 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 12 Sep 2018 02:37:05 GMT
< Connection: keep-alive
< 
{
  "2_ahS8BAG2s": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-v01.api.letsencrypt.org left intact

$ curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 23.9.148.127...
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:79e::3a8e...
* Immediate connect fail for 2001:8006:3510:79e::3a8e: Cannot assign requested address
*   Trying 2001:8006:3510:788::3a8e...
* Immediate connect fail for 2001:8006:3510:788::3a8e: Cannot assign requested address
* Connected to acme-v02.api.letsencrypt.org (23.9.148.127) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 596 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: acme-v02.api.letsencrypt.org (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=acme-v02.api.letsencrypt.org
* 	 start date: Fri, 03 Aug 2018 01:36:30 GMT
* 	 expire date: Thu, 01 Nov 2018 01:36:30 GMT
* 	 issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /directory HTTP/1.1
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 658
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Wed, 12 Sep 2018 02:39:03 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Wed, 12 Sep 2018 02:39:03 GMT
< Connection: keep-alive
< 
{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "wdvW-gzlrGc": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
* Connection #0 to host acme-v02.api.letsencrypt.org left intact

politas · September 12, 2018, 5:00am

I have solved the issue. This was caused by my server having two NICs active on the same subnet. Moving one to a different VLAN has got all working again.

system · October 12, 2018, 5:10am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.