HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Read timed out

Help
1

During the renewal process I get the error message:

Renewing an existing certificate
Attempting to renew cert (irish-wolfhound-of-lough-ree.de) from /etc/letsencrypt/renewal/irish-wolfhound-of-lough-ree.de.conf produced an unexpected error: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Read timed out. (read timeout=45). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/irish-wolfhound-of-lough-ree.de/fullchain.pem (failure)

What to do?

Volker

2

Hello. I’ve moved your post to a new topic (with an excessively long title).

Can you post the complete output from Certbot?

Can you also look for the error in /var/log/letsencrypt/letsencrypt.log (or wherever the log is), post the traceback, and the last few lines before it?

There could be something wrong with your connectivity to https://acme-v02.api.letsencrypt.org/, but also the service is probably under high load today, so random errors like that are probably going to happen sometimes. :grimacing:

Does it work if you try again?

Does that certificate have an especially large number of (sub)domains?

Does “curl -v https://acme-v02.api.letsencrypt.org/directory” work? What does it output?

Can you also fill out the rest of the questionnaire below?

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

1 Like
3

There is also a currently ongoing “Security Issue”:
https://letsencrypt.status.io/
image

image
image919×190 5.59 KB

4

Hello,

here are my answers to your questions:

My domain is: admin.irish-wolfhound-of-lough-ree.de, irish-wolfhound-of-lough-ree.de, www.irish-wolfhound-of-lough-ree.de

I ran this command: certbot renew --force-renewal

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/irish-wolfhound-of-lough-ree.de.conf

Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (irish-wolfhound-of-lough-ree.de) from /etc/letsencrypt/renewal/irish-wolfhound-of-lough-ree.de.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/irish-wolfhound-of-lough-ree.de/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/irish-wolfhound-of-lough-ree.de/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

My web server is (include version): Server version: Apache/2.4.18 (Ubuntu) , Server built: 2019-10-08T13:31:25

The operating system my web server runs on is (include version): Linux meerkat 4.15.0-88-generic #88~16.04.1-Ubuntu SMP Wed Feb 12 04:19:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is: self hosted

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

The contents of the log file is:

Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f60c619fda0>
2020-03-03 12:02:41,929:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2020-03-03 12:02:41,929:DEBUG:certbot.renewal:no renewal failures
2020-03-04 06:55:41,898:DEBUG:certbot.main:certbot version: 0.31.0
2020-03-04 06:55:41,899:DEBUG:certbot.main:Arguments: ['-q']
2020-03-04 06:55:41,900:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
"/var/log/letsencrypt/letsencrypt.log" 4034 lines, 275692 characters
chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 388, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 452, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1193, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 310, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 369, in obtain_certificate
cert, chain = self.obtain_certificate_from_csr(csr, orderr)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 301, in obtain_certificate_from_csr
orderr = self.acme.finalize_order(orderr, deadline)
File "/usr/lib/python3/dist-packages/acme/client.py", line 927, in finalize_order
return self.client.finalize_order(orderr, deadline)
File "/usr/lib/python3/dist-packages/acme/client.py", line 754, in finalize_order
self._post(orderr.body.finalize, wrapped_csr)
File "/usr/lib/python3/dist-packages/acme/client.py", line 96, in _post
return self.net.post(*args, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1204, in post
return self._post_once(*args, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1217, in _post_once
response = self._send_request('POST', url, data=data, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1120, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 502, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 612, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

2020-03-04 09:51:09,885:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2020-03-04 09:51:09,885:ERROR:certbot.renewal: /etc/letsencrypt/live/irish-wolfhound-of-lough-ree.de/fullchain.pem (failure)
2020-03-04 09:51:09,885:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
return config.func(config, plugins)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1272, in renew
renewal.handle_renewal_request(config)
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 477, in handle_renewal_request
len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)

The output of the curl command is:

  • Trying 2606:4700:60:0:f53d:5624:85c7:3a2c...
  • Trying 172.65.32.248...
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • found 148 certificates in /etc/ssl/certs/ca-certificates.crt
  • found 592 certificates in /etc/ssl/certs
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  •    server certificate verification OK

  •    server certificate status verification SKIPPED

  •    common name: acme-v01.api.letsencrypt.org (matched)

  •    server certificate expiration date OK

  •    server certificate activation date OK

  •    certificate public key: RSA

  •    certificate version: #3

  •    subject: CN=acme-v01.api.letsencrypt.org

  •    start date: Fri, 07 Feb 2020 02:19:13 GMT

  •    expire date: Thu, 07 May 2020 02:19:13 GMT

  •    issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3

  •    compression: NULL
  • ALPN, server accepted to use http/1.1

GET /directory HTTP/1.1
Host: acme-v02.api.letsencrypt.org
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 04 Mar 2020 09:03:17 GMT
< Content-Type: application/json
< Content-Length: 658
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
<
{
"FmDmNTw2qP4": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"

Best regards,
Volker

5

I'm not sure if Certbot is unable to connect to https://acme-v02.api.letsencrypt.org/, or if it successfully connects and then it breaks later.

It looks like curl tries to use IPv6, it fails, and then it uses IPv4, which works.

Does your server have IPv6 connectivity? Does it work?

1 Like
6

Yes, the server supports IPv6. Just to be sure I have deactivated it and now 'curl' reports:

  • Trying 172.65.32.248...
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • found 148 certificates in /etc/ssl/certs/ca-certificates.crt
  • found 592 certificates in /etc/ssl/certs
  • ALPN, offering http/1.1
  • SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
  •    server certificate verification OK

  •    server certificate status verification SKIPPED

  •    common name: acme-v01.api.letsencrypt.org (matched)

  •    server certificate expiration date OK

  •    server certificate activation date OK

  •    certificate public key: RSA

  •    certificate version: #3

  •    subject: CN=acme-v01.api.letsencrypt.org

  •    start date: Sun, 12 Jan 2020 18:06:08 GMT

  •    expire date: Sat, 11 Apr 2020 18:06:08 GMT

  •    issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3

  •    compression: NULL
  • ALPN, server accepted to use http/1.1

GET /directory HTTP/1.1
Host: acme-v02.api.letsencrypt.org
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 04 Mar 2020 09:33:11 GMT
< Content-Type: application/json
< Content-Length: 658
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
<
{
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"nG9MF72FHEU": "Adding random entries to the directory",
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"

Nevertheless the renewal still reports an error:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/irish-wolfhound-of-lough-ree.de.conf

Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (irish-wolfhound-of-lough-ree.de) from /etc/letsencrypt/renewal/irish-wolfhound-of-lough-ree.de.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/irish-wolfhound-of-lough-ree.de/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/irish-wolfhound-of-lough-ree.de/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

7

Have you checked to ensure DNS is working properly?

8

Yes, nslookup gives me

Server: 127.0.1.1
Address: 127.0.1.1#53

Non-authoritative answer:
acme-v02.api.letsencrypt.org canonical name = prod.api.letsencrypt.org.
prod.api.letsencrypt.org canonical name = ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
Name: ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 172.65.32.248

9

127.0.1.1 is a local address…
Perhaps (at times) the local system is having issues with name resolution or running low on resources or …
Can you (temporarily) switch to another DNS server?
Can you show a screenshot of top ?

10

Ok, I have changed the nameserver to 8.8.8.8, but the behaviour is unchanged; the command still ends up with “Read timed out”. Here’s the request screenshot of the top command:

grafik
grafik605×551 7.31 KB

11

Definitely NOT an internal resource problem.

  • 77.7% unused mem (never used)
  • 95.3% free mem
  • 99.9% CPU idle
  • 0.00% swapfile use [you could probably just turn that off: swapoff -a]

So that still leaves testing via another DNS server...

Like:
8.8.8.8, 8.8.4.4
1.1.1.1, 1.0.0.1
208.67.222.220, 208.67.222.222
4.2.2.2, 4.2.2.4

12

That's also expected on systems running systemd-resolved :wink:

13

As mentioned before I switched over to 8.8.8.8 as DNS server.

14

Hi @VSiebelink

what says

traceroute acme-v02.api.letsencrypt.org

Perhaps reduce your MTU to 1300 or 1100. Sometimes that helps.

15

traceroute to ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248), 64 hops max
1 10.11.12.1 0,505ms 0,530ms 0,527ms
2 62.156.244.49 63,755ms 11,780ms 11,738ms
3 62.156.247.178 12,415ms 12,130ms 12,171ms
4 217.0.195.205 15,478ms 15,112ms 15,236ms
5 217.0.195.205 15,084ms 14,749ms 14,830ms
6 62.157.249.186 15,034ms 14,950ms 14,935ms
7 * * *
8 129.250.4.187 15,313ms 15,186ms 15,393ms
9 213.198.81.142 15,559ms 17,347ms 15,622ms
10 * * *
11 * * *
12 * * *
13 * * *

‘ping’ is working without any problems.

I don’t think that it is a network issue or something like that, because the automatic renewal of the certificates works for more than a year without any problems.

16

Looks like this is a part of your problem.

traceroute / tracert should work.

D:\temp>tracert -4 acme-v02.api.letsencrypt.org.

Routenverfolgung zu ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com [172.65.32.248]
über maximal 30 Hops:

1 <1 ms <1 ms <1 ms fritz.box [192.168.0.1]
2 4 ms 5 ms 5 ms 62.155.240.117
3 7 ms 6 ms 6 ms 217.239.55.2
4 6 ms 5 ms 6 ms 217.239.55.2
5 7 ms 6 ms 6 ms lag-10.edge4.Berlin1.Level3.net [4.68.73.5]
6 7 ms 6 ms 6 ms ae-1-3502.edge3.Berlin1.Level3.net [4.69.159.1]
7 6 ms 6 ms 8 ms unknown.Level3.net [212.162.40.34]
8 6 ms 5 ms 6 ms 172.65.32.248

Last year Letsencrypt switched to another CDN solution.

A lot of topics with such connection problems, reducing MTU sometimes helped.

17

Also, Certbot might be trying to use IPv6.

18

Hi @JuergenAuer

thank you very much for insisting on changing the MTU! Indeed this was the cause of the problem. After reducing it to 1300 the renewal ran without any complains :slight_smile:

Best regards,
Volker

2 Likes
19

Ah, happy to read it had worked. Thanks for reporting back :+1:

1 Like
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.