Letsencrypt and Nginx fails acme challenge

My domain is:
occamsmachete.com

I ran this command:
sudo certbot --authenticator webroot --installer nginx

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer nginx

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: occamsmachete.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for occamsmachete.com
Input the webroot for occamsmachete.com: (Enter 'c' to cancel): /home/pat/occamsmachete.com/
Waiting for verification...
Challenge failed for domain occamsmachete.com
http-01 challenge for occamsmachete.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: occamsmachete.com
   Type:   connection
   Detail: Fetching
   http://occamsmachete.com/.well-known/acme-challenge/b25nbc8A5dZA9HP46VT8WSM2OgkYTkSiCjGPQuvnDx8:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version):
nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 20.04

My hosting provider, if applicable, is:
Self hosted

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 1.7.0
from Ubuntu snap certbot “classic”

The acme-challenge fails.
I have /home/pat/occamsmachete.com/.well-known/acme-challenge mapped to /.well-known/acme-challenge with a dummy html file at http://occamsmachete.com/.well-known/acme-challenge/dummy.html This is accessible from curl and browsers. So the acme challenge path seems to be writable via the filesystem and accessible via http.

I have nginx setup to host a node.js blog but allow the challenge path to also be available:

server {
        listen 80;
        server_name occamsmachete.com;
	root /home/pat/occamsmachete.com;
	index index.html;

	location / {
		proxy_pass	http://127.0.0.1:2368;
	        proxy_set_header    X-Real-IP $remote_addr;
	        proxy_set_header    Host      $http_host;
		proxy_set_header X-Forwarded-Proto https;
	        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

	}

	location /.well-known/acme-challenge {
		default_type "text/plain";
		root /home/pat/occamsmachete.com/;
	}
}

certbot reads the nginx config and knows my domain name. The domain is hosted on enom and points to my server as can be seen by following the URL. Certbot asks for my webroot, which I enter as “/home/occamsmachete.com” but fails as shown above with “Connection refused” log attached.

letsencrypt.log here: https://gist.github.com/pferrel/cf1320513b58c5007e470675991c5bb2

Not sure what I’m doing wrong but I’ve tried many variations and am being blocked temporarily.

I could connect to your server moments ago, but now I see:

$ curl -i occamsmachete.com
curl: (7) Failed to connect to occamsmachete.com port 80: Connection refused

Did you stop your server?

more experimenting, sorry, try again, please :slightly_smiling_face:

Still down from here.

Maybe that is the problem. I am running from a machine in my closet. I have a Comcast GB broadband router with ports forwarded to make it all work.

From the server: pat@angel : ~ $ curl occamsmachete.com
some html here…

I’m pretty sure some kind of firewall is involved.

Initially, I could connect from both my home internet connection and from letsdebug.net.

But after a few connections, my laptop and letsdebug.net can no longer connect to your server.

Then, I connected from yet a another server, and I could connect twice. Third time, I’m now getting connection refused.

Tried from another server, and was able to connect 5 times, and then got locked out again.

Here’s the last one:

root@syd-01:~# curl -X GET -I occamsmachete.com
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sun, 23 Aug 2020 00:27:58 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 7805
Connection: keep-alive
X-Powered-By: Express
Cache-Control: public, max-age=0
ETag: W/"1e7d-p+1nkL4pKYb4Hobj+3hCuC4HPzk"
Vary: Accept-Encoding

root@syd-01:~# curl -X GET -I occamsmachete.com
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sun, 23 Aug 2020 00:28:00 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 7805
Connection: keep-alive
X-Powered-By: Express
Cache-Control: public, max-age=0
ETag: W/"1e7d-p+1nkL4pKYb4Hobj+3hCuC4HPzk"
Vary: Accept-Encoding

root@syd-01:~# curl -X GET -I occamsmachete.com
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sun, 23 Aug 2020 00:28:01 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 7805
Connection: keep-alive
X-Powered-By: Express
Cache-Control: public, max-age=0
ETag: W/"1e7d-p+1nkL4pKYb4Hobj+3hCuC4HPzk"
Vary: Accept-Encoding

root@syd-01:~# curl -X GET -I occamsmachete.com
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sun, 23 Aug 2020 00:28:02 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 7805
Connection: keep-alive
X-Powered-By: Express
Cache-Control: public, max-age=0
ETag: W/"1e7d-p+1nkL4pKYb4Hobj+3hCuC4HPzk"
Vary: Accept-Encoding

root@syd-01:~# curl -X GET -I occamsmachete.com
curl: (7) Failed to connect to occamsmachete.com port 80: Connection refused
root@syd-01:~# curl -X GET -I occamsmachete.com
curl: (7) Failed to connect to occamsmachete.com port 80: Connection refused
root@syd-01:~# curl -X GET -I occamsmachete.com
curl: (7) Failed to connect to occamsmachete.com port 80: Connection refused
root@syd-01:~# curl -X GET -I occamsmachete.com
curl: (7) Failed to connect to occamsmachete.com port 80: Connection refused

There was once a user here who discovered that their router had a really crude “firewall” function which had a similar effect. I’d look into that. Or if you use fail2ban or something like it on your server, I’d try disable that.

Very odd but that would explain my problem.

I have Ubuntu 20.04 a clean install and have used ufw to enable ssh, http, and https. So I seriously doubt it is my server itself. Maybe something to do with Comcast—grrr.

Thanks for the debug info.

BTW if you have dig can you check what it returns there? I have tried from phones connected only via the carrier and it works but haven’t tried over and over.

dig consistently produces 73.35.185.91, I don’t think it’s a DNS problem.

You can try use https://letsdebug.net/occamsmachete.com/266808?debug=y to see the same info and re-run the tests.

If your modem/router does have a web UI, I would very much suggest double checking that there’s no firewall stuff enabled in there.

Being a Comcast/Xfinity residential user, i personally didn't find any blocks put in place by them on HTTP/HTTPS ports. If you want to further debug that, can you try to hard-wire your server to router and allow me to do a port test? (It can generally tell me if it's not listening, filtered or listening)

A small piece of thought: Is it possible that the mutiple requests Let's Encrypt initiated flooded the server?

1 Like

Funny you should mention that. Based on the behavior you all have been observing, I was reminded of another issue posted in Help the other day that ultimately was caused by a strict firewall rule based on multiple connection attempts that was blocking the Let's Encrypt server from verifying http challenges. This might be the same case that @_az was referencing.

The case I was remembering was from the start of 2019: Can't upgrade certbot on Mac OS 10.7

1 Like

I was referring to this one:

Now that I look back at it, I realize that @stevenzhu is already more than familiar with it. :upside_down_face:

1 Like

I have double checked all firewalls involved and have disabled any (for now). If anyone still sees connection problems, I value the information – many thanks. From comments I take it there are tools that I could use to check connections but I can’t get to the sites referenced, can someone repeat the suggestions?

More info:

I have a Comcast Gigabit router/modem connected to a Netgear Orbi connected to my home hosted server. The settings for all devices seem to allow all connections without restrictions (see below).

Comcast uses DHCP to manage my IP address but no changes have been made by them since I put the server up. I’m working on a realtime updater to use DDNS but don’t think it is the problem as yet since no change to my IP address has been made.

I don’t have a good way to verify what others are seeing, namely intermittent ability to connect. Many thanks for any info others might have regarding connections or suggestions to investigate.

Comcast router:

  • I just completely disabled the firewall on this device, it was only blocking IDENT port 113 previously.
  • ports forwarded to Orbi device: 22, 80, 443

Orbi router:

  • ports forwarded to my server via a static reserved address: 22, 80, 443
  • no other firewall settings.

Server:

  • ubuntu 20.04

  • ufw status:
    Status: active

    To Action From


    22/tcp ALLOW Anywhere
    80/tcp ALLOW Anywhere
    443/tcp ALLOW Anywhere
    22/tcp (v6) ALLOW Anywhere (v6)
    80/tcp (v6) ALLOW Anywhere (v6)
    443/tcp (v6) ALLOW Anywhere (v6)

I have a Comcast Gigabit router/modem connected to my Orbi LAN router connected to my server. Are you asking me to hard wire my server to the Comcast router? They are physically separated but I could do that…

I have an AWS machine for other purposes but don’t have root login at present. Once I get this I can port scan myself but as I said in the thread reply it certainly looks all clear from the firewall perspective. There are no number of connection restrictions that I am aware of.

How did you get your server connected to the internet?

If anyone cares to try now that all firewalls are down:

http://occamsmachete.com
http:73.35.185.91

Hi @pferrel

there are only timeouts - see https://check-your-website.server-daten.de/?q=occamsmachete.com

There is an older check, 16.08.2020 23:20:43 - with answers.

Something blocks.

Mine is a Synology NAS at home and hard wired to one of the Wifi Routers (Mesh setup). I'm not using an Xfinity router or their gateway but I know their gateway sometimes has connection restrictions.

Maybe it's time to give Xfinity a call? Perhaps they have some restriction on your account or router. (Just random thought)

P.S. Both port 80 and 443 are shown as filtered from my port test.

OMG, thanks for that link the report is here: https://check-your-website.server-daten.de/?i=9397fab8-e0eb-4780-a723-21ca5f6bcb21 What an awesome tool! It says letsencrypt cannot succeed, which we now know is something to do with making connections, as expected.

There are issues with my DNS server’s TLD and IPv6 addresses for my domain, which I think I understand but don’t seem to be urgent problems.

However it reports this: “Fatal error: Nameserver isn’t defined or has timeout” I don’t understand this since the address is resolved, which means the DNS server returned the IP address for my domain, right?

If you have any thoughts about the primary problem this report contains it might help.

For my part I have used ssh to login to 2 AWS machines in US zones and they are able to curl to occamsmachete.com repeatedly. Also dig seems to report the correct IP being resolved, from any machine I’ve tried.

Can I try any linux tool (from AWS machine) to give similar info? I have used a lot of linux networking tools but so far have not seen the kind of info as on the server-daten.de site.

Thanks very much, it seems your help is narrowing down the problem considerably.

More info:

trying nc on my web server in question.

**pat@angel** : **~** $ nc -zv 192.168.1.128 22 80 443
Connection to 192.168.1.128 22 port [tcp/ssh] succeeded!
Connection to 192.168.1.128 80 port [tcp/http] succeeded!
nc: connect to 192.168.1.128 port 443 (tcp) failed: Connection refused

**pat@angel** : **~** $ nc -zv occamsmachete.com 22 80 443
Connection to occamsmachete.com 22 port [tcp/ssh] succeeded!
Connection to occamsmachete.com 80 port [tcp/http] succeeded!
nc: connect to occamsmachete.com port 443 (tcp) failed: Connection refused

I do not have my server listening to 443 yet, is this a problem? There is no cert yet.

Some of your name servers send the correct answer.

But others fail. Authoritative name servers must support tcp connections, that's an IANA-requirement.

Not answering name servers are always bad. But you can't change these, you can only change your dns provider.

And your website doesn't answer.