Standalone certbot renew error: connection reset by peer (timeout)

I have an FTP server that needs a certificate to perform FTPS connections. I set up certbot a while ago to get certificates for the server using the Standalone method, since there wasn’t a web server on this machine I needed to worry about. Port 80 is open to the machine specifically for certificate renewals. However, a few months ago, when trying to renew the certificate, it would sit on the “Waiting for verification…” stage for some time, then throw a timeout exception.

Can someone help me track down what might be going wrong with this renewal process? FTP connections to the server are working fine, but apparently HTTP requests to the server aren’t going through during the renewal process. I can port scan the IP while the renewal is happening and it detects that port 80 is open on the machine, but requests just end up timing out.


My domain is: ftp.altalang.com

I ran this command: sudo certbot renew --force-renewal

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ftp.altalang.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ftp.altalang.com
Waiting for verification...
Cleaning up challenges
----------------------------------------
Exception happened during processing of request from ('::ffff:52.28.236.88', 26590, 0, 0)
Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 320, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3/dist-packages/acme/standalone.py", line 206, in __init__
    BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
  File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
    self.handle()
  File "/usr/lib/python3/dist-packages/acme/standalone.py", line 215, in handle
    BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python3.6/http/server.py", line 418, in handle
    self.handle_one_request()
  File "/usr/lib/python3.6/http/server.py", line 386, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------
Attempting to renew cert (ftp.altalang.com) from /etc/letsencrypt/renewal/ftp.altalang.com.conf produced an unexpected error: Failed authorization procedure. ftp.altalang.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://ftp.altalang.com/.well-known/acme-challenge/pws10BMqkHVInpoGJZOAdYufl7rudAMyl2SY3BBwxKA: Timeout after connect (your server may be slow or overloaded). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ftp.altalang.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ftp.altalang.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: /etc/letsencrypt/renewal-hooks/post/restart-proftpd.sh
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: ftp.altalang.com
   Type:   connection
   Detail: Fetching
   http://ftp.altalang.com/.well-known/acme-challenge/pws10BMqkHVInpoGJZOAdYufl7rudAMyl2SY3BBwxKA:
   Timeout after connect (your server may be slow or overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): None – this is just an FTP server (ProFTPD)

The operating system my web server runs on is (include version): Ubuntu 18.04 LTS

My hosting provider, if applicable, is: N/A (using our own hardware)

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.31.0

I have not had the chance to review your post fully and confirm. There is a chance that your results today are related to load issues Let’s Encrypt is experiencing. We have asked subscribers to renew approximately 3 million certificates in preparation for revocations related to 2020.02.29 CAA Rechecking Bug

Hopefully, someone with more certbot knowledge will be able to help you out soon. Thanks for your patience while we respond to the influx of help requests on the forum today.

1 Like

I’d say this is either a load issue (on Let’s Encrypt’s end due to the crush of renewals) or a firewall issue (on the subscriber end with something that interrupts some inbound connections on port 80). There is a slight possibility of the latter because the ProFTPD service mentions it’s on the internal address 10.0.4.65, which is different from the public address 107.7.87.202. So it’s possible that there’s a router or firewall forwarding ports which is dropping the connections for some reason, based on some characteristic.

2 Likes

Possibly, but I believe I was running into these same issues last month. I’m only posting about it now because I went back to revisit the issue after the announcement, and it seems to be showing the same issues as before.

My understanding of the error message was that a remote server was timing out trying to contact my server, i.e. it is trying to make an HTTP GET request to http://ftp.altalang.com/.well-known/acme-challenge/pws10BMqkHVInpoGJZOAdYufl7rudAMyl2SY3BBwxKA and timing out. Since port 80 is only accepting requests when renewing the certificate, and since that URL changes every time certbot renew runs, I decided to test the HTTP request myself. I ran sudo certbot renew -v --dry-run to enable verbose logging, and snagged the URL from the output logs, and pasted it into my browser while the renew ran to see if it would respond, and it never responded. So I’m still feeling like this might be an issue on my end, but it’s certainly possible I’ve made a bad assumption, here.

1 Like

Update: I think it’s working now, without changing anything on my end, so perhaps it was due to the load issues after all. However, it’s still giving me an exception:

$ sudo certbot renew --force-renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ftp.altalang.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ftp.altalang.com
Waiting for verification...
Cleaning up challenges
----------------------------------------
Exception happened during processing of request from ('::ffff:52.28.236.88', 19590, 0, 0)
Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 320, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3/dist-packages/acme/standalone.py", line 206, in __init__
    BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
  File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
    self.handle()
  File "/usr/lib/python3/dist-packages/acme/standalone.py", line 215, in handle
    BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python3.6/http/server.py", line 418, in handle
    self.handle_one_request()
  File "/usr/lib/python3.6/http/server.py", line 386, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/ftp.altalang.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/ftp.altalang.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: /etc/letsencrypt/renewal-hooks/post/restart-proftpd.sh

I see my new cert expires on June 3rd so it definitely worked, but I don’t know what’s up with that Python exception still.

These behaviors still make me think of a firewall somewhere interrupting these connections for some reason.