Challenge verification is failing on requests error: requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(104, 'ECONNRESET')"))

Hello,
we have created our own certbot image for docker with running cronjobs for our wildcard certificates.
We are using dns-01 method and we have manual-auth-hook for changing DNS entries on our DNS provider side.
It was working correctly, but after some time - two months ago it started failing on challenge verification where there should be POST to challenge URL to invoke DNS check.

In debug mode it tells me what it wants to POST where.
When I run requests in that container in python with same data/headers/url it works correctly and challenge status is changed from pending to valid.

I am trying to figure this problem out already for second week and it’s starting to be critical as our certificates expire in two days and infrastructure of our company will blow up…

Sending raw exception stacktrace. Tried to format it but nothing helped…

Encountered exception:
Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1321, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 296, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 312, in recv_into
return self.recv_into(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 302, in recv_into
raise SocketError(str(e))
OSError: (104, ‘ECONNRESET’)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File “/usr/lib/python3.7/site-packages/urllib3/util/retry.py”, line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/usr/lib/python3.7/site-packages/urllib3/packages/six.py”, line 685, in reraise
raise value.with_traceback(tb)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1321, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 296, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 312, in recv_into
return self.recv_into(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 302, in recv_into
raise SocketError(str(e))
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, OSError("(104, ‘ECONNRESET’)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/certbot/auth_handler.py”, line 87, in handle_authorizations
self.acme.answer_challenge(achall.challb, resp)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 150, in answer_challenge
response = self._post(challb.uri, response)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 96, in _post
return self.net.post(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 1189, in post
return self._post_once(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 1202, in _post_once
response = self._send_request(‘POST’, url, data=data, **kwargs)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 1103, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/usr/lib/python3.7/site-packages/requests/sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python3.7/site-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python3.7/site-packages/requests/adapters.py”, line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, OSError("(104, ‘ECONNRESET’)"))

Calling registered functions
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1321, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 296, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 312, in recv_into
return self.recv_into(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 302, in recv_into
raise SocketError(str(e))
OSError: (104, ‘ECONNRESET’)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File “/usr/lib/python3.7/site-packages/urllib3/util/retry.py”, line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/usr/lib/python3.7/site-packages/urllib3/packages/six.py”, line 685, in reraise
raise value.with_traceback(tb)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1321, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 296, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 312, in recv_into
return self.recv_into(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 302, in recv_into
raise SocketError(str(e))
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, OSError("(104, ‘ECONNRESET’)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/sbin/certbot”, line 11, in
load_entry_point(‘certbot==0.36.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3.7/site-packages/certbot/main.py”, line 1381, in main
return config.func(config, plugins)
File “/usr/lib/python3.7/site-packages/certbot/main.py”, line 1264, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/usr/lib/python3.7/site-packages/certbot/main.py”, line 115, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/usr/lib/python3.7/site-packages/certbot/renewal.py”, line 307, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File “/usr/lib/python3.7/site-packages/certbot/client.py”, line 349, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3.7/site-packages/certbot/client.py”, line 385, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/usr/lib/python3.7/site-packages/certbot/auth_handler.py”, line 87, in handle_authorizations
self.acme.answer_challenge(achall.challb, resp)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 150, in answer_challenge
response = self._post(challb.uri, response)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 96, in _post
return self.net.post(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 1189, in post
return self._post_once(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 1202, in _post_once
response = self._send_request(‘POST’, url, data=data, **kwargs)
File “/usr/lib/python3.7/site-packages/acme/client.py”, line 1103, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/usr/lib/python3.7/site-packages/requests/sessions.py”, line 533, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python3.7/site-packages/requests/sessions.py”, line 646, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python3.7/site-packages/requests/adapters.py”, line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, OSError("(104, ‘ECONNRESET’)"))
An unexpected error occurred:
Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1321, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 296, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 312, in recv_into
return self.recv_into(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 302, in recv_into
raise SocketError(str(e))
OSError: (104, ‘ECONNRESET’)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.7/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File “/usr/lib/python3.7/site-packages/urllib3/util/retry.py”, line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/usr/lib/python3.7/site-packages/urllib3/packages/six.py”, line 685, in reraise
raise value.with_traceback(tb)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 600, in urlopen
chunked=chunked)
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 384, in _make_request
six.raise_from(e, None)
File “”, line 2, in raise_from
File “/usr/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 380, in _make_request
httplib_response = conn.getresponse()
File “/usr/lib/python3.7/http/client.py”, line 1321, in getresponse
response.begin()
File “/usr/lib/python3.7/http/client.py”, line 296, in begin
version, status, reason = self._read_status()
File “/usr/lib/python3.7/http/client.py”, line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 312, in recv_into
return self.recv_into(*args, **kwargs)
File “/usr/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 302, in recv_into
raise SocketError(str(e))
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, OSError("(104, ‘ECONNRESET’)"))

During handling of the above exception, another exception occurred:

requests.exceptions.ConnectionError: (‘Connection aborted.’, OSError("(104, ‘ECONNRESET’)"))

Hi @MrIncognito,

Can you provide the domain you’re attempting to issue for so I can check logs to see if you’re even making a successful connection to the API?

Can you also run the following commands from inside the container and on the container host?

ping acme-v02.api.letsencrypt.org
curl https://acme-v02.api.letsencrypt.org/directory
mtr -c 20 acme-v02.api.letsencrypt.org --report

# IPv6
ping -6 acme-v02.api.letsencrypt.org
curl -6 https://acme-v02.api.letsencrypt.org/directory
mtr -6 -c 20 acme-v02.api.letsencrypt.org --report
2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.