Certbot exiting with error: ConnectionError: ('Connection aborted.', error("(60, 'ETIMEDOUT')",))

Help
1

Hi,

I ran this command:

sudo certbot certonly --manual

I created, deployed (to google app engine) and ensured both challenge files were served at the appropriate urls:
http://www.tailorcv.biz/.well-known/acme-challenge/Kyot9TV6KbAB9E-zg-seGp5A3JiqOW5KYy7sDp52Is4

http://tailorcv.biz/.well-known/acme-challenge/uShRtzln0mxmxI8npStVcD42FQzZUQ5yyzk9MuRw5Y0

certbot exited with this:

Waiting for verification...
An unexpected error occurred:
ConnectionError: ('Connection aborted.', error("(60, 'ETIMEDOUT')",))
Please see the logfiles in /var/log/letsencrypt for more details.

The log file had this:

017-05-21 05:58:22,369:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/local/bin/certbot", line 11, in
load_entry_point('certbot==0.14.0', 'console_scripts', 'certbot')()
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/main.py", line 742, in main
return config.func(config, plugins)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/main.py", line 682, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/main.py", line 82, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/client.py", line 344, in obtain_and_enroll_certificate
certr, chain, key, _ = self.obtain_certificate(domains)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/client.py", line 313, in obtain_certificate
self.config.allow_subset_of_names)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/auth_handler.py", line 81, in get_authorizations
self._respond(resp, best_effort)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/auth_handler.py", line 134, in _respond
resp, chall_update)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/certbot/auth_handler.py", line 158, in _send_responses
self.acme.answer_challenge(achall.challb, resp)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/acme/client.py", line 229, in answer_challenge
response = self.net.post(challb.uri, response)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/acme/client.py", line 674, in post
return self._post_once(*args, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/acme/client.py", line 685, in _post_once
response = self._send_request('POST', url, data=data, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/acme/client.py", line 619, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error("(60, 'ETIMEDOUT')",))

I was getting the same error when I tried getting certificates yesterday but I thought that was because of the outage.

Thanks for any help.

2

Hi @tcv

Your files looks good.

I think the timeout is in comms to the Let’s Encrypt API

Can you try

wget https://acme-v01.api.letsencrypt.org/directory

Andrei

3

Hi Andrei,

Thanks for your reply. Here is the output of the wget command:

➜ /tmp wget https://acme-v01.api.letsencrypt.org/directory
--2017-05-22 08:47:52-- https://acme-v01.api.letsencrypt.org/directory
Resolving acme-v01.api.letsencrypt.org... 104.96.99.79, 2001:559:19:609a::3d5, 2001:559:19:6090::3d5
Connecting to acme-v01.api.letsencrypt.org|104.96.99.79|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 352 [application/json]
Saving to: ‘directory’

directory 100%[=====================================================================================================>] 352 --.-KB/s in 0s

2017-05-22 08:47:53 (16.0 MB/s) - ‘directory’ saved [352/352]

4

Based on this reply it seems like the issue might be resolved. Are you still seeing the reported timeout error using Certbot to issue a certificate?

5

I was able to run the wget command, am am now in the process of rerunning the certbot command to see if it will work. I will update this thread with status.

Thanks

1 Like
6

The timeout error is still occurring:

Press Enter to Continue
Waiting for verification...
An unexpected error occurred:
ConnectionError: ('Connection aborted.', error("(60, 'ETIMEDOUT')",))
Please see the logfiles in /var/log/letsencrypt for more details.
➜ /tmp sudo tail /var/log/letsencrypt/letsencrypt.log
Password:
response = self._send_request('POST', url, data=data, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/acme/client.py", line 619, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/Cellar/certbot/0.14.0/libexec/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error("(60, 'ETIMEDOUT')",))
➜ /tmp

7

hi @tcv

Looks like it could be a python issue

run> pip freeze and you should get a list of packages

can you also try this snippet of code and let me know what you get

python
import requests
r = requests.get('https://acme-v01.api.letsencrypt.org/directory')
r.status_code
r.text

Should be something like the below

image.png974×233 14.5 KB

Andrei

8

Hi Andrei,

I don't have much in my system python:

➜ ~ pip freeze
ipaddress==1.0.18
python-dateutil==2.6.0
six==1.10.0
virtualenv==15.1.0

I don't have requests in system python but if I install it in a virtualenv the code works:

(venv) ➜ requests python
Python 2.7.13 (default, Apr 4 2017, 08:46:44)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import requests
r = requests.get('https://acme-v01.api.letsencrypt.org/directory')
r.status_code
200
r.text
u'{\n "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",\n "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",\n "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",\n "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",\n "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"\n}'

Maybe I've got a bad version/install of certbot? I was thinking of uninstalling and reinstalling certbot from my mac or maybe installing certbot from a debian vm i have and trying from there.

Thanks

9

hi @tcv

A reinstall would probably be the best way forward

Andrei

10

Hi Andrei,

I uninstalled certbot from my mac and removed /etc/letsencrypt and reinstalled and am now on version 0.14.1.

I’m still getting the same error.

I’m wondering if this error is happening because of the time it takes to deploy the challenge file to google app engine. I timed it and from the time i’m presented with the challenge to the time the challenge file is available at the challenge url, it’s about 10 minutes. That doesn’t seem like that long but I thought I’d ask.
Thanks,
Perviz

11

This timeout is not a timeout related to something that you’re doing, but in attempting to connect over HTTPS to the Let’s Encrypt CA. That’s why it’s strange that wget appears to work fine but Certbot doesn’t—they’re trying to connect to the same server over HTTPS.

What if you used curl instead of wget for that test?

Do you have IPv6 as well as IPv4 connectivity for your own computer? Does your computer think you have IPv6 as well as IPv4 connectivity?

12

Thanks for the clarification.

curl gives the same result as wget:

/tmp curl https://acme-v01.api.letsencrypt.org/directory
{
"key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
"new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
"new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
"new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
"revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}%

According to ifconfig and http://test-ipv6.com/ I do not have an ipv6 address:

/tmp ifconfig en0 | grep inet6
inet6 fe80::9a01:a7ff:fe99:e915%en0 prefixlen 64 scopeid 0x4

13

I tried certbot-auto on a Debian VM and was able to get a cert for my domain. I’m going to use this cert for now but I’d like to fix the issue with certbot on my mac because this is my development machine and I just tried the Debian VM to validate my usage of certbot.

Any pointers you can provide to debug the certbot installation on my mac are much appreciated.

Thanks for all your help!

14

I think this is very bizarre, because the code that @ahaw021 had you run is doing something very similar to the code that timed out, and even using the same underlying library to try to access almost the same Internet resource.

If you want to keep debugging this, I guess we would want to modify your copy of Certbot by adding more logging statements in Python so that we can get more details about what it’s doing.

15

For kicks, I cloned GitHub - certbot/certbot: Certbot is EFF's tool to obtain certs from Let's Encrypt and (optionally) auto-enable HTTPS on your server. It can also act as a client for any other CA that uses the ACME protocol. so I could see if I could reproduce the error from a development version. I ran certbot from docker-compose and also got an abnormal exit (not the same error, but in about the same place):

File "/opt/certbot/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
resp = self.send(prep, **send_kwargs)
File "/opt/certbot/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
r = adapter.send(request, **kwargs)
File "/opt/certbot/venv/local/lib/python2.7/site-packages/requests/adapters.py", line 488, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error("(104, 'ECONNRESET')",))

I'm kind of surprised this isn't working for me when running from docker as well (maybe I shouldn't be).

Does the connection error being ECONNRESET instead of ETIMEDOUT (like it was in my brew installed certbot) provide any clues?

So I can either debug the brew installed certbot on my mac or the certbot development version. Let me know which one would be more helpful.

Thanks.

16

I guess the development version… it’s really rather peculiar!

Do you have any other tools that you could use to confirm whether you really don’t have an IPv6 connection? Maybe try

echo -e 'GET / HTTP/1.0\n\n' | nc -6 acme-v01.api.letsencrypt.org 80 echo -e 'GET / HTTP/1.0\n\n' | nc -4 acme-v01.api.letsencrypt.org 80

17

Here is the output of both commands:

➜ certbot git:(master) echo -e 'GET / HTTP/1.0\n\n' | nc -6 acme-v01.api.letsencrypt.org 80
➜ certbot git:(master) echo -e 'GET / HTTP/1.0\n\n' | nc -4 acme-v01.api.letsencrypt.org 80
HTTP/1.0 400 Bad Request
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 208
Expires: Fri, 26 May 2017 19:27:22 GMT
Date: Fri, 26 May 2017 19:27:22 GMT
Connection: close

Invalid URL

Invalid URL

The requested URL "[no URL]", is invalid.

Reference #9.de32c517.1495826842.1858515 ➜ certbot git:(master)

18

OK, how about @ahaw021’s

python
import requests
r = requests.get('https://acme-v01.api.letsencrypt.org/directory')
r.status_code
r.text

(which you already tried before, but in this environment)?

19

When you say 'in this environment, I take it to mean from docker shell:

root@f761a5e3cf7a:/opt/certbot/src# which python
/opt/certbot/venv/bin/python
root@f761a5e3cf7a:/opt/certbot/src# python
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import requests
r = requests.get('https://acme-v01.api.letsencrypt.org/directory')
r.status_code
200
r.text
u'{\n "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",\n "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",\n "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",\n "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",\n "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"\n}'

20

I added retry logic to certbot/acme/acme/client.py in _send_request:

    # retry 10 times:
    logger.debug('--retrying 10 times...')
    for i in range(10):
        logger.debug('... %s', i)
        try:
            response = self.session.request(method, url, *args, **kwargs)
            logger.debug(' --worked, breaking out...')
            break
        except Exception as e:
            logger.debug('-- got this error: %s', e)
    logger.debug('--out of for loop, this is response: %s', response)

I was expecting this to fail 10 times but it worked on the first try. Here is the debug log:

2017-05-26 20:27:08,770:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2017-05-26 20:27:08,770:DEBUG:acme.client:--retrying 10 times...
2017-05-26 20:27:08,770:DEBUG:acme.client:... 0
2017-05-26 20:27:08,771:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-05-26 20:27:08,955:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 352
2017-05-26 20:27:08,956:DEBUG:acme.client: --worked, breaking out...
2017-05-26 20:27:08,956:DEBUG:acme.client:--out of for loop, this is response: <Response [200]>
2017-05-26 20:27:08,956:DEBUG:acme.client:Received response:
HTTP 200

It's great that it worked on the first try, but I have no idea why it would.