Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/git.biome.io.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for git.biome.io
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (git.biome.io) from /etc/letsencrypt/renewal/git.biome.io.conf produced an unexpected error: Failed authorization procedure. git.biome.io (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://git.biome.io/.well-known/acme-challenge/LSdQt_Gxp-EAWl0zTxYOqJKptCFsha-IgehjZpatqVY: Connection reset by peer. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/git.biome.io/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/git.biome.io/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: git.biome.io
Type: connection
Detail: Fetching
http://git.biome.io/.well-known/acme-challenge/LSdQt_Gxp-EAWl0zTxYOqJKptCFsha-IgehjZpatqVY:
Connection reset by peer
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you're using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.
My web server is (include version): nginx via gitlab
The operating system my web server runs on is (include version): Ubuntu 18.04.6 LTS
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0
Perhaps another volunteer with more gitlab experience will offer specific advice. In the meantime, your site is not responding to any requests with HTTP (or HTTPS). Once you get HTTP working again the challenge by Let's Encrypt should work too. Example
curl -iv http://git.biome.io
* Trying 13.83.15.185:80...
* connect to 13.83.15.185 port 80 failed: Connection refused
* Failed to connect to git.biome.io port 80 after 74 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to git.biome.io port 80 after 74 ms: Connection refused
Looks like there are different things acting on incoming connections:
A firewall blocking to all well known ports (FTP, SSH, SMTP, POP3, IMAP), except port 80 and 443. So that firewall is set up correctly it seems;
Those incoming connections on port 80 and/or 443 are directed to a host which is not running any service on those ports. Perhaps a misconfigured NAT router?
I see you got a cert today and it is being sent out by your server. So that's great.
But, renewals won't work with the setup you have now. You should test it with
certbot renew --dry-run
You may have a firewall blocking requests specifically of the ACME challenge path. Or maybe your redirects are doing something odd (proxying to a downstream server?).
Look carefully at these sample curl responses.
curl -I git.biome.io/.well-known/acme-challeng
(challenge not fully spelled out)
HTTP/1.1 404 Not Found
Server: nginx
Date: Tue, 19 Apr 2022 22:00:40 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
curl -I git.biome.io/.well-known/acme-challenge
(no trailing slash, redirects fine but adds trailing slash)
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 19 Apr 2022 21:50:05 GMT
Content-Type: text/html
Content-Length: 162
Location: http://git.biome.io/.well-known/acme-challenge/
Connection: keep-alive
As soon as the slash appears the request fails.
curl -I git.biome.io/.well-known/acme-challenge/
curl: (56) Recv failure: Connection reset by peer
ForumTest would be the name of the ACME challenge file for actual requests
This should respond with 404 (not found) as ForumTest file does not exist
curl -I git.biome.io/.well-known/acme-challenge/ForumTest
curl: (56) Recv failure: Connection reset by peer
Hi - thank you so much for the suggestions. We've been working feverishly to get this back up & running and like you point out, we did eventually get a certificate through a roundabout process -- we cloned the VM, brought a copy up in a different network environment, renewed our certificates, and then brought the renewed snapshot back into the original environment -- but the original environment is not set up correctly.
I noticed the curl response oddities as well, and it seemed to be a firewall issue but we could not find the actual problem. It's an Ubuntu VM (we checked iptables and ufw, both seemed fine, and we disabled SentinelOne for the purposes of our test) running on Azure (we checked the network gateway there, as well as Azure firewalls) and managed by a third party which runs a Palo Alto firewall on top of everything else. We'll need to figure out exactly where the issue is in order to renew for next time.
This was additionally complicated for us by the fact that the nginx instance in question is packaged with an older version of gitlab, and the nginx config file and logs were not in the usual location.
If we find another lead or the solution we will make sure to post it here, and thank you again for your help!
We just experienced the same "Connection reset by peer" issue, where previously renewal has worked perfectly, and upon checking our Palo Alto firewall it's recently started blocking the acme-protocol "application" in its settings with the default deny rule. So I'd suggest checking your firewall in case the same has occured.
Thanks very much for your post. We have been seeing these regularly the past couple weeks. Yours is the first that posted a vendor and setting. Will be a good place to start when we see the next one.