LE API 'Connection reset by peer'

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

What is strange is that the next command that ran in the script worked just fine and generate the script for the next domain needed.

My domain is:

wkweiea9ixstat.mgddns.io

I ran this command:


2021-10-27 11:03:47,354:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1514/bin/certbot
2021-10-27 11:03:47,354:DEBUG:certbot._internal.main:Arguments: ['--webroot', '-w', '/var/www/wkweiea9ixstat.mgddns.io/htdocs', '--cert-name', 'wkweiea9ixstat.mgddns.io', '-d', 'wkweiea9ixstat.mgddns.io', '--email', 'hostservices@mgdwp.com', '--no-eff-email', '--agree-tos', '--staple-ocsp', '--must-staple', '--preferred-chain', 'ISRG Root X1', '--preconfigured-renewal']

It produced this output:


2021-10-27 11:03:47,354:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-10-27 11:03:47,373:DEBUG:certbot._internal.log:Root logging level set at 30
2021-10-27 11:03:47,374:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2021-10-27 11:03:47,379:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: Authenticator, Plugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7fd5d2cebb80>
Prep: True
2021-10-27 11:03:47,380:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot.Authenticator object at 0x7fd5d2cebb80> and installer None
2021-10-27 11:03:47,380:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2021-10-27 11:03:47,389:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/256890810', new_authzr_uri=None, terms_of_service=None), 563eea62195134f2517cdee7f18ce9f1, Meta(creation_dt=datetime.datetime(2021, 10, 27, 10, 57, 36, tzinfo=<UTC>), creation_host='localhost', register_to_eff=None))>
2021-10-27 11:03:47,390:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2021-10-27 11:03:47,393:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2021-10-27 11:04:02,591:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/snap/certbot/1514/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/snap/certbot/1514/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/snap/certbot/1514/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/certbot/1514/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/connection.py", line 411, in connect
    self.sock = ssl_wrap_socket(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/snap/certbot/1514/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/snap/certbot/1514/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/snap/certbot/1514/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/snap/certbot/1514/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/snap/certbot/1514/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/1514/lib/python3.8/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/1514/lib/python3.8/site-packages/certbot/_internal/main.py", line 1574, in main
    return config.func(config, plugins)
  File "/snap/certbot/1514/lib/python3.8/site-packages/certbot/_internal/main.py", line 1416, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/snap/certbot/1514/lib/python3.8/site-packages/certbot/_internal/main.py", line 770, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
  File "/snap/certbot/1514/lib/python3.8/site-packages/certbot/_internal/client.py", line 262, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
  File "/snap/certbot/1514/lib/python3.8/site-packages/certbot/_internal/client.py", line 44, in acme_from_config_key
    client = acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/snap/certbot/1514/lib/python3.8/site-packages/acme/client.py", line 840, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/snap/certbot/1514/lib/python3.8/site-packages/acme/client.py", line 1194, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/snap/certbot/1514/lib/python3.8/site-packages/acme/client.py", line 1133, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/snap/certbot/1514/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/snap/certbot/1514/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/snap/certbot/1514/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2021-10-27 11:04:02,603:ERROR:certbot._internal.log:An unexpected error occurred:
2021-10-27 11:04:02,604:ERROR:certbot._internal.log:requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

My web server is (include version):

nginx version: nginx/1.16.1

The operating system my web server runs on is (include version):
Ubuntu 20.04.3 LTS
My hosting provider, if applicable, is:

Private cloud

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.20.0

2

@techpad Can you connect to other URLs from your server?

Try:
curl -I https://acme-v02.api.letsencrypt.org/directory
to check the failing connection without using Certbot and python.

And, try:

curl -4 https://ifconfig.co

To test basic connectivity

3 Likes
3

For completeness, please show the output for all four cases:
curl -4 http://ifconfig.co/
curl -4 https://ifconfig.co/
curl -6 http://ifconfig.co/
curl -6 https://ifconfig.co/

3 Likes
4

@rg305
root@wpms-shost1:~# curl -4 http://ifconfig.co/
102.129.237.40
root@wpms-shost1:~# curl -4 https://ifconfig.co/
102.129.237.40
root@wpms-shost1:~# curl -6 http://ifconfig.co/
curl: (7) Couldn't connect to server
root@wpms-shost1:~# curl -6 https://ifconfig.co/
curl: (7) Couldn't connect to server

2 Likes
5

curl -I https://acme-v02.api.letsencrypt.org/directory
HTTP/2 200
server: nginx
date: Fri, 29 Oct 2021 17:32:51 GMT
content-type: application/json
content-length: 658
cache-control: public, max-age=0, no-cache
replay-nonce: 00028RK5drHJvuXFtIcoa5hw_hHjr4avD2IJEa1TKm5AzSo
x-frame-options: DENY
strict-transport-security: max-age=604800

2 Likes
6

@techpad I am just poking in hopes of something becoming clearer. So far nothing seems odd except the failed connect you showed in original post.

Can you try your original Certbot command (certonly I am guessing) but adding --dry-run as an option? That will do as you tried but against the Lets Encrypt test "staging" servers. Curious to see if those fail in same way.

And, show the actual command - not just the parse results from the log. Thanks

2 Likes
7

@JamesLE, please check this IP is not being blocked - thanks.

2 Likes
8

Are the same IPs blocked by both staging and production servers? I was assuming staging would not block but now I am not sure.

3 Likes
9

I don't know.
I would presume that they are both behind the same protection system - but that is just my assumption.

3 Likes
10

I'm looking to determine the exact command used. What is weird is there were two domains. This first one failed, the one right after worked.

1 Like
11

I also know of another client who's run into this. If more logs would help I could ask them to join this thread.

1 Like
12

Hmmm. If another worked from same IP that rules out LE blocking that IP. Bummer I was hoping that was it :slight_smile:

So, does the failing one you showed fail repeatedly? Or was it a one-off?

Can you show the original commands from both the working and failed? Or, at least the log from the one that worked. In /var/log/letsencrypt it keeps a history of prior attempts.

Do you run the commands yourself or is there some sort of shell, wrapper, or interface used to run certbot?

2 Likes
13

It’s a one off on this server. That said, I'm starting to see these random failures across my servers. When I find a little more free time I'm going to collect the errors that I've seen and I'll post any that are having the same issue. I know some of these fail for random reasons, but this has happened twice in the last week on new server builds on unused IP's in my blocks.

2 Likes
14

@techpad Sounds good on gathering more info. Not sure if this is helpful but more poking I see these items:

crt.sh shows you got a cert 2 days ago.

Yet, the https response for that server returns a self-signed cert:

curl -I https://wkweiea9ixstat.mgddns.io
curl: (60) SSL certificate problem: self signed certificate

Also see sslshopper.com/sslchecker.html

Trying -k to bypass verification gives a failure that it should not:

curl -Ik https://wkweiea9ixstat.mgddns.io
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

This is starting to "feel" like a comms issue at your end but debugging that is getting beyond my area of expertise.

3 Likes
15

We're not blocking this IP address.

4 Likes
16

Yeah, this is far from a standard config.
[OR the MiTM has been exposed! - LOL]

3 Likes
17

I've actually left this as is to troubleshoot. This is setting up an endpoint to connect to a Monit stats page protected by NGNIX http password. The entire script bailed out at this error so it's an incomplete configuration at this point.

1 Like
18

This is the command that was run from the script.

certbot-auto certonly --webroot \
  -w /var/www/wkweiea9ixstat.mgddns.io/htdocs \
  --cert-name "wkweiea9ixstat.mgddns.io" \
  -d "wkweiea9ixstat.mgddns.io" \
  --email hostservices@mgdwp.io --no-eff-email --agree-tos \
  --staple-ocsp --must-staple \
  --preferred-chain "ISRG Root X1" \
  2>&1

I just ran it and it worked without any issue.

19

@rg305 & @MikeMcQ

I went back to the let's encrypt logs and i pasted all the rotated logs in order and I think I'm seeing a trend. I would like to post the whole thing, but it's spitting out everything including the certs. Besides having to rerun the cert or even just rebuilding the server. Is posting this whole script a no no on this forum or is it OK.

I suspect the script that is building this might be hammering the API, or maybe the server is running the script quickly and it's hammering the API. In any event I'd love a second set of eyes on this.

1 Like
20

@techpad Before posting huge logs, there looks to be a problem with the command you used. You show certbot-auto but that was deprecated in favor of just certbot. This was primarily due to it needing python2.

Yet, your initial log showed a snap install of certbot with python3. This is the "modern" install. And, you noted certbot version 1.20 which is recent. I do not understand the migration of that older install method well but Rudy and others will be better able to help sort this out. Let's see what they have to say.

Here is a post about that from Dec 2020:

And, the current install page that mentions it too:
https://certbot.eff.org/docs/install.html#certbot-auto

4 Likes