Read timeout error

xiviu123 · February 28, 2018, 4:08am

curl https://acme-v01.api.letsencrypt.org -I
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html
Content-Length: 2174
Last-Modified: Fri, 02 Feb 2018 23:36:17 GMT
ETag: "5a74f5f1-87e"
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Accept-Ranges: bytes
Expires: Wed, 28 Feb 2018 04:11:06 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 28 Feb 2018 04:11:06 GMT
Connection: keep-alive

I ran this command: /usr/local/bin/certbot-auto --apache certonly

It produced this output: Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter ‘c’ to
cancel):hhh@gmail.com
An unexpected error occurred:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py”, line 391, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py”, line 387, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib64/python3.4/http/client.py”, line 1227, in getresponse
response.begin()
File “/usr/lib64/python3.4/http/client.py”, line 386, in begin
version, status, reason = self._read_status()
File “/usr/lib64/python3.4/http/client.py”, line 348, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib64/python3.4/socket.py”, line 378, in readinto
return self._sock.recv_into(b)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/contrib/pyopenssl.py”, line 271, in recv_into
raise timeout(‘The read operation timed out’)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/adapters.py”, line 423, in send
timeout=timeout
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py”, line 643, in urlopen
_stacktrace=sys.exc_info()[2])
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/util/retry.py”, line 334, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/packages/six.py”, line 686, in reraise
raise value
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py”, line 594, in urlopen
chunked=chunked)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py”, line 393, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py”, line 313, in _raise_timeout
raise ReadTimeoutError(self, url, “Read timed out. (read timeout=%s)” % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=‘acme-v01.api.letsencrypt.org’, port=443): Read timed out. (read timeout=45)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 9, in
load_entry_point(‘letsencrypt==0.7.0’, ‘console_scripts’, ‘letsencrypt’)()
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 1240, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 1104, in certonly
le_client = _init_le_client(config, auth, installer)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 632, in _init_le_client
acc, acme = _determine_account(config)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 511, in _determine_account
config, account_storage, tos_cb=_tos_cb)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/client.py”, line 165, in register
regr = perform_registration(acme, config)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/client.py”, line 195, in perform_registration
return acme.register(messages.NewRegistration.from_data(email=config.email))
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/acme/client.py”, line 98, in register
response = self.net.post(self.directory[new_reg], new_reg)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/acme/client.py”, line 709, in post
return self._post_once(*args, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/acme/client.py”, line 720, in _post_once
response = self._send_request(‘POST’, url, data=data, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/acme/client.py”, line 630, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/sessions.py”, line 488, in request
resp = self.send(prep, **send_kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/sessions.py”, line 609, in send
r = adapter.send(request, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/requests/adapters.py”, line 499, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host=‘acme-v01.api.letsencrypt.org’, port=443): Read timed out. (read timeout=45)
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version): ache/2.2.15

The operating system my web server runs on is (include version): centos 6.8

thank for help

_az · February 28, 2018, 4:19am

Please run

host acme-v01.api.letsencrypt.org
curl -X GET -I --connect-timeout 10 -H "Pragma: akamai-x-cache-on, akamai-x-get-cache-key, akamai-x-get-true-cache-key, akamai-x-get-request-id, akamai-x--meta-trace, akama-xi-get-extracted-values, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-cache-remote-on, akamai-x-get-client-ip" https://acme-v01.api.letsencrypt.org/
traceroute acme-v01.api.letsencrypt.org

It may be the case that the timeout problem only occurs on POST requests, I saw this happening previously, which ultimately resolved itself but we never tracked the cause down.

There is a possible workaround to override the IP address of the API server in /etc/hosts. If you want, you could try point the domain to 23.49.216.139 or another location, and see if Certbot succeeds against that.

mnordhoff · February 28, 2018, 4:29am

It could also potentially be an IPv6 issue.

Does the curl command run quickly?

Does the computer have IPv6?

If it is using IPv6 – or trying to – using “traceroute6” may be necessary.

xiviu123 · February 28, 2018, 5:04am

my computer using ipv4 not v6

traceroute acme-v01.api.letsencrypt.org

traceroute to acme-v01.api.letsencrypt.org (23.76.67.207), 30 hops max, 60
byte packets

1 203.162.166.1 (203.162.166.1) 1.931 ms 1.932 ms 2.390 ms

2 localhost (123.31.44.115) 0.244 ms 0.283 ms 0.275 ms

3 static.vnpt.vn (123.29.5.134) 0.508 ms 0.403 ms 0.534 ms

4 static.vnpt.vn (113.171.5.197) 25.649 ms 25.704 ms *

5 static.vnpt.vn (123.29.6.50) 21.863 ms static.vnpt.vn
(113.171.34.26) 29.505
ms 29.454 ms

6 static.vnpt.vn (113.171.44.74) 25.720 ms 31.412 ms 26.100 ms

7 Bundle-Ether20.931.br02.hkg08.pccwbtn.net (63.217.66.145) 71.310
ms 72.154
ms 71.491 ms

8 HundredGE0-6-0-1.br04.hkg15.pccwbtn.net (63.223.17.90) 67.062 ms * 64.368
ms

9 TenGE0-2-0-4.br04.hkg15.pccwbtn.net (63.223.15.102) 64.015 ms
TenGE0-5-0-1.br04.hkg15.pccwbtn.net (63.223.15.186) 66.001 ms 62.319 ms

10 ix-ae-22-0.tcore1.HK2-Hong-Kong.as6453.net (180.87.112.93) 39.079
ms 34.845
ms 39.093 ms

11 if-ae-7-2.thar1.HK2-Hong-Kong.as6453.net (180.87.112.142) 39.212
ms 39.147
ms 38.390 ms

12 116.0.82.158 (116.0.82.158) 38.560 ms 38.361 ms 39.285 ms

13 a23-76-67-207.deploy.static.akamaitechnologies.com (23.76.67.207) 39.823
ms 35.214 ms 39.185 ms

Stay the Same

xiviu123 · February 28, 2018, 5:07am

thanks for help, after add host ploblem is resolved . Many thank to you

_az · February 28, 2018, 5:15am

Vietnam?

The previous user with the exact same issue was also from Vietnam: ERROR about Issue SSL let's Encrypt in cPanel , which was never solved.

@xiviu123 could you please run the diagnostics shown in this post, after undoing the workaround. The /etc/hosts is not a safe long-term fix, because the IP address may change without any notice, since it is managed by Akamai.

schoen · February 28, 2018, 5:59am

Maybe we could mention to @jillian that this issue now appears to have come back again in Vietnam.

_az · February 28, 2018, 7:43am

Well, another customer from VN just emailed me with the same complaint . I am not sure if this is coincidental timing or whether some network configuration has shifted again.

schoen · February 28, 2018, 11:53pm

@jillian @isk @jsha, it looks like we might have an Akamai problem going on in Vietnam at the moment.

_az · March 1, 2018, 2:58am

I got my hands on a VPS in VN but unfortunately it doesn’t exhibit the issue. DM me if you want access to the VPS, it’s not used for anything.

If we look at the trace from this problem-free VPS:

# traceroute acme-v01.api.letsencrypt.org
traceroute to acme-v01.api.letsencrypt.org (23.76.67.207), 30 hops max, 60 byte packets
1  gateway (103.199.16.1)  0.389 ms  0.829 ms  0.972 ms
2  125.212.199.189 (125.212.199.189)  5.663 ms  5.894 ms  6.234 ms
3  103.1.208.9 (103.1.208.9)  0.731 ms  0.756 ms  0.772 ms
4  localhost (27.68.225.21)  0.706 ms  0.735 ms  0.803 ms
5  localhost (27.68.229.217)  33.610 ms  33.625 ms  33.613 ms
6  localhost (27.68.228.9)  39.991 ms  38.758 ms  39.660 ms
7  localhost (27.68.240.26)  32.623 ms  32.694 ms localhost (27.68.240.66)  33.039 ms
8  63-217-254-249.static.pccwglobal.net (63.217.254.249)  33.152 ms  33.071 ms  33.278 ms
9  HundredGE0-7-0-0.br04.hkg15.pccwbtn.net (63.223.15.202)  33.967 ms HundredGE0-6-0-1.br

and compare it to OP’s problematic server:

# traceroute acme-v01.api.letsencrypt.org
traceroute to acme-v01.api.letsencrypt.org (23.76.67.207), 30 hops max, 60 byte packets
1 203.162.166.1 (203.162.166.1) 1.931 ms 1.932 ms 2.390 ms
2 localhost (123.31.44.115) 0.244 ms 0.283 ms 0.275 ms
3 static.vnpt.vn (123.29.5.134) 0.508 ms 0.403 ms 0.534 ms
4 static.vnpt.vn (113.171.5.197) 25.649 ms 25.704 ms *
5 static.vnpt.vn (123.29.6.50) 21.863 ms static.vnpt.vn (113.171.34.26) 29.505ms 29.454ms
6 static.vnpt.vn (113.171.44.74) 25.720 ms 31.412 ms 26.100 ms
7 Bundle-Ether20.931.br02.hkg08.pccwbtn.net (63.217.66.145) 71.310 ms 72.154ms 71.491 ms
8 HundredGE0-6-0-1.br04.hkg15.pccwbtn.net (63.223.17.90) 67.062 ms * 64.368 ms
9 TenGE0-2-0-4.br04.hkg15.pccwbtn.net (63.223.15.102) 64.015 ms
TenGE0-5-0-1.br04.hkg15.pccwbtn.net (63.223.15.186) 66.001 ms 62.319 ms
10 ix-ae-22-0.tcore1.HK2-Hong-Kong.as6453.net (180.87.112.93) 39.079ms 34.845ms 39.093 ms
11 if-ae-7-2.thar1.HK2-Hong-Kong.as6453.net (180.87.112.142) 39.212ms 39.147ms 38.390 ms
12 116.0.82.158 (116.0.82.158) 38.560 ms 38.361 ms 39.285 ms
13 a23-76-67-207.deploy.static.akamaitechnologies.com (23.76.67.207) 39.823ms 35.214 ms 39.185 ms

and the issue from January (reverse trace, as I don’t have access to that server):

traceroute to 103.57.222.10 (103.57.222.10), 30 hops max, 60 byte packets
1  27.131.68.137 (27.131.68.137)  1.236 ms  1.067 ms  1.120 ms
2  111.223.226.129 (111.223.226.129)  0.551 ms  0.689 ms  0.875 ms
3  192.168.253.66 (192.168.253.66)  0.183 ms 192.168.253.65 (192.168.253.65)  0.178 ms  0.207 ms
4  192.168.249.62 (192.168.249.62)  0.252 ms 192.168.249.58 (192.168.249.58)  0.861 ms 192.168.249.62 (192.168.249.62)  0.842 ms
5  port-channel10.win22.melbourne.telstra.net (110.145.148.205)  1.635 ms  0.807 ms  1.590 ms
6  bundle-ether3-100.win-core10.melbourne.telstra.net (203.50.80.129)  1.586 ms  3.695 ms  3.405 ms
7  bundle-ether6.fli-core1.adelaide.telstra.net (203.50.11.90)  11.164 ms  11.087 ms  12.350 ms
8  bundle-ether5.wel-core3.perth.telstra.net (203.50.11.19)  38.785 ms  41.056 ms  40.955 ms
9  tengige0-8-0-5-1.pie-core1.perth.telstra.net (203.50.6.198)  41.146 ms  41.110 ms  41.604 ms
10  203.50.9.2 (203.50.9.2)  101.288 ms  101.335 ms  101.306 ms
11  i-0-4-2-0.sgpl-core01.bx.telstraglobal.net (202.84.143.94)  126.384 ms  126.396 ms  126.370 ms
12  i-0-6-0-4.hkth-core01.bx.telstraglobal.net (202.84.144.189)  214.345 ms i-0-3-0-1.hkth-core01.bx.telstraglobal.net (202.84.136.145)  215.062 ms i-0-2-0-8.hkth-core01.bx.telstraglobal.net (202.84.141.137)  215.050 ms
13  i-0-0-0-18.hkth12.bi.telstraglobal.net (202.84.156.9)  213.630 ms i-0-0-0-0.hkth12.bi.telstraglobal.net (202.84.153.141)  214.732 ms i-0-4-0-18.hkth12.bi.telstraglobal.net (202.84.156.13)  217.289 ms
14  tenge4-2.br01.hkg08.pccwbtn.net (63.218.205.145)  133.841 ms  133.761 ms  137.944 ms
15  TenGE0-1-0-5.br03.hkg15.pccwbtn.net (63.218.210.138)  132.338 ms HundredGE0-5-0-0.br03.hkg15.pccwbtn.net (63.223.17.162)  158.604 ms TenGE0-1-0-5.br03.hkg15.pccwbtn.net (63.218.210.138)  132.921 ms
16  HundredGE0-5-0-0.br03.hkg15.pccwbtn.net (63.223.17.162)  162.958 ms HundredGE0-5-0-1.br03.hkg15.pccwbtn.net (63.223.15.210)  133.721 ms HundredGE0-5-0-0.br03.hkg15.pccwbtn.net (63.223.17.162)  158.449 ms
17  63-218-211-166.static.pccwglobal.net (63.218.211.166)  303.639 ms  299.046 ms  299.290 ms
18  static.vnpt.vn (113.171.5.202)  325.974 ms static.vnpt.vn (113.171.33.82)  324.235 ms static.vnpt.vn (113.171.5.202)  327.690 ms
19  static.vnpt.vn (113.171.33.46)  327.912 ms  327.776 ms  328.011 ms
20  10.132.132.44 (10.132.132.44)  322.230 ms  326.453 ms  321.737 ms
21  nethost-1511.inet.vn (103.57.222.10)  327.713 ms  327.989 ms  327.925 ms

We see that the common culprit in the two problematic cases is vnpt.vn.

So I would hypothesize is that there is a problem between vnpt.vn and the HK Akamai presence, but not between vnpt.vn and other (out-of-region) Akamai POPs.

HTH

system · March 31, 2018, 2:58am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot - URLLIB Read Timeout Help	7	2235	July 3, 2017
ReadTimeout: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45) Help	13	3456	March 24, 2021
Acme-v02.api.letsencrypt.org - Read timed out Help	3	1666	January 8, 2022
ReadTimeout: HTTPSConnectionPool(host='acme-v01.api.letsencrypt.org', port=443): Read timed out. (read timeout=45) Help	1	1651	February 8, 2018
ReadTimeout: HTTPSConnectionPool(host='acme-v01.api.letsencrypt.org', port=443) Help	31	24782	May 19, 2017

Read timeout error

Related topics