Renewal failure to due apparent hang during connection

robinzyx · March 23, 2020, 9:30pm

My domain is: [redacted]

I ran this command: /path/certbot-auto renew

It produced this output: See logging appended below.

My web server is (include version): Apache 2.4

The operating system my web server runs on is (include version): CentOS 6

My hosting provider, if applicable, is: –

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 1.3

It is time to renew the Let’s Encrypt SSL cert on my office website. We’ve been using Let’s Encrypt for a couple years without trouble, but with this renewal, it is total failure.

When certbot-auto was manually run last week to renew the cert, it automatically updated to 1.3 from the prior 1.0 or 1.1 install that was in place. (The last cert renewal occurred in mid-January.) Thereafter the process simply hangs and the certificate is not updated. Eventually a ctrl-C is required. A copy of the letsencrypt.log from a dry-run is appended below.

I am in the process of trying to see if I can restore our copy of letsencrypt/certbot to an older version obtained from backup, and then apply the --no-self-upgrade option. But if that doesn’t work, and also for future updates’ sake, I’d like to figure out how to fix this and renew our SSL cert.

Thx for any and all help.

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/[redacted].conf

^CExiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
load_entry_point(‘letsencrypt==0.7.0’, ‘console_scripts’, ‘letsencrypt’)()
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/main.py”, line 15, in main
return internal_main.main(cli_args)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/main.py”, line 1347, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/main.py”, line 1255, in renew
renewal.handle_renewal_request(config)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/renewal.py”, line 433, in handle_renewal_request
if should_renew(lineage_config, renewal_candidate):
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/renewal.py”, line 264, in should_renew
if lineage.should_autorenew():
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/storage.py”, line 946, in should_autorenew
if self.ocsp_revoked(self.latest_common_version()):
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/storage.py”, line 906, in ocsp_revoked
chain_path)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/ocsp.py”, line 100, in ocsp_revoked_by_paths
return _check_ocsp_cryptography(cert_path, chain_path, url)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/ocsp.py”, line 168, in _check_ocsp_cryptography
headers={‘Content-Type’: ‘application/ocsp-request’})
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/api.py”, line 116, in post
return request(‘post’, url, data=data, json=json, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/api.py”, line 60, in request
return session.request(method=method, url=url, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py”, line 672, in urlopen
chunked=chunked,
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py”, line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1036, in _send_output
self.send(msg)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 974, in send
self.connect()
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connection.py”, line 184, in connect
conn = self._new_conn()
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connection.py”, line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/util/connection.py”, line 74, in create_connection
sock.connect(sa)
KeyboardInterrupt
Please see the logfiles in /var/log/letsencrypt for more details.

_az · March 23, 2020, 11:42pm

It looks like an OCSP request timeout.

Are you filtering outbound traffic from your server? HTTP traffic to http://ocsp.int-x3.letsencrypt.org must be permitted. (For staging on the current intermediate, it’s http://ocsp.stg-int-x1.letsencrypt.org).

Nonetheless, this feel like a Certbot bug to me, because there is no timeout set in the OCSP querying code, which results in a potential infinite hang. I have filed an issue for it: https://github.com/certbot/certbot/issues/7859

robinzyx · March 24, 2020, 12:34am

Yes, some sort of OCSP timeout request going on. See below.

One of the IT people at my office asked how long I had been letting it sit in the hang state. I couldn’t give him a specific value, so I decided, what the heck let’s try it again and let it sit for a really long time.

It turned out that a dry-run apparently finished successfully in about 15 minutes. So I tried the actual renewal, and that was also apparently successful, but it took 26 minutes. In the past, the renewal would take no longer than a minute, maybe two at most.

I say apparently because there were several Python warnings/errors before Certbot finished off. See below for what I got doing the renewal. But the server does now have a new cert with an expiration date in June rather than in April. Whew.

As far as I know, there is no filtering by domain of the outgoing traffic. However, the server is in a sort of DMZ for security reasons and it’s possible that there has been a configuration change in the firewall within the past two months. I will have to query the IT people a couple steps up from my office people.

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/[redacted].conf

OCSP check failed for /etc/letsencrypt/archive/[redacted]/cert15.pem (are we offline?)
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connection.py”, line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/util/connection.py”, line 84, in create_connection
raise err
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/util/connection.py”, line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py”, line 672, in urlopen
chunked=chunked,
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py”, line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 1036, in _send_output
self.send(msg)
File “/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py”, line 974, in send
self.connect()
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connection.py”, line 184, in connect
conn = self._new_conn()
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connection.py”, line 169, in _new_conn
self, “Failed to establish a new connection: %s” % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f29fa30eda0>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py”, line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/urllib3/util/retry.py”, line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=‘ocsp.int-x3.letsencrypt.org’, port=80): Max retries exceeded with url: / (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x7f29fa30eda0>: Failed to establish a new connection: [Errno 110] Connection timed out’,))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/ocsp.py”, line 168, in _check_ocsp_cryptography
headers={‘Content-Type’: ‘application/ocsp-request’})
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/api.py”, line 116, in post
return request(‘post’, url, data=data, json=json, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/api.py”, line 60, in request
return session.request(method=method, url=url, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/requests/adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host=‘ocsp.int-x3.letsencrypt.org’, port=80): Max retries exceeded with url: / (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x7f29fa30eda0>: Failed to establish a new connection: [Errno 110] Connection timed out’,))
Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
http-01 challenge for [redacted]
Waiting for verification…
Cleaning up challenges

new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/[redacted]/fullchain.pem

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/[redacted]/fullchain.pem (success)

_az · March 24, 2020, 12:44am

Thanks for adding that info.

From what you can tell, all the certificates would have renewed eventually as part of the automated cron, albeit very slowly, right? You did not have to make any other interventions?

robinzyx · March 24, 2020, 1:39am

@_az, It seems we have belatedly figured out the issue.

It appears that in a recent update to certbot, there is an attempt to communicate with ocsp.int-x3.letsencrypt.org:80 that has been added?

The problem was that since our server is in a DMZ, the HTTPS and HTTP proxies need to be set before executing certbot. However, the configuration we had for calling certbot had been only setting the HTTPS proxy and not the HTTP. Previously that did not cause any trouble, but this time around, the attempts to reach the above address were hanging.

After updating the config, I just ran a renewal dry run and it executed in normal time.

_az · March 24, 2020, 1:41am

Yup: https://github.com/certbot/certbot/blob/master/certbot/CHANGELOG.md#130---2020-03-03

(OCSP checks are done over HTTP because there's a chicken-and-egg problem with verifying the certificate chain if you can't check if the certificates in the chain are revoked ... or something).

system · April 23, 2020, 1:41am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.