Cert renew/creation fails on stock Ubuntu

This story starts with me receiving an email the certificates for my domain can not be renewed. I tried to renew manually and got the error message listed below (timeout during connect (likely firewall problem)).
I have been using certbot for years now and have extensive host/networking experience.
I simplified everything and even reinstalled nginx/certbot, removed the certificates and tried to recreate them but still the same error.
I tried 'telnet www.janvanoorschot.nl 80' and got a correct reply from nginx. I tried to create the '.well-known' file manually and then copied the exact URL from the error message and paste it in my browser and it worked. I also remote ssh'ed in a friendly host and did a 'wget' for the same file and it worked.
I also did a 'watch -n 1 ls -al' in the .well-known' directory during the certbot operation but saw no file being created.

Can anybody please help? I am running out of idea's.

Thanks! ... Jan

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
janvanoorschot.nl
I ran this command:
sudo certbot --nginx -d janvanoorschot.nl -d www.janvanoorschot.nl

It produced this output:
Requesting a certificate for janvanoorschot.nl and www.janvanoorschot.nl

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: janvanoorschot.nl
Type: connection
Detail: 178.250.146.51: Fetching http://janvanoorschot.nl/.well-known/acme-challenge/oUenbzWFaYFGNui54ycy09vj91wDhamEsuYqgjx7XrI: Timeout during connect (likely firewall problem)

Domain: www.janvanoorschot.nl
Type: connection
Detail: 178.250.146.51: Fetching http://www.janvanoorschot.nl/.well-known/acme-challenge/YMwEBhxAPKUclrEesRqDuEJy5G7A05pk5babLuUXFKE: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

My web server is (include version):
nginx ( nginx-core (1.18.0-6ubuntu14.3)
The operating system my web server runs on is (include version):
Ubuntu 22.04

My hosting provider, if applicable, is:
not applicable
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.4.0

1 Like

Hello @Janvanoorschot, welcome to the Let's Encrypt community. :slightly_smiling_face:

Any chance the is a strange Firewall, possible Palo Alto Firewall, in your network?

You are using HTTP-01 challenge of the Challenge Types - Let's Encrypt, the requires Port 80 be open and accessible. Best Practice - Keep Port 80 Open

Using Let's Debug yields test results https://letsdebug.net/janvanoorschot.nl/1414557?debug=y

Using nmap -Pn I see nothing of significance
This one seems strange to me.

deleted

And this one looks normal to me.

$ nmap -Pn janvanoorschot.nl
Starting Nmap 7.80 ( https://nmap.org ) at 2023-03-20 14:55 UTC
Nmap scan report for janvanoorschot.nl (178.250.146.51)
Host is up (0.15s latency).
rDNS record for 178.250.146.51: dev.robomindacademy.com
Not shown: 995 closed ports
PORT    STATE    SERVICE
25/tcp  filtered smtp
80/tcp  open     http
135/tcp filtered msrpc
139/tcp filtered netbios-ssn
445/tcp filtered microsoft-ds

Nmap done: 1 IP address (1 host up) scanned in 40.61 seconds
1 Like

Also the traceroute hints to me of a routing loop; 17 & 18 are the same.

$ sudo traceroute -T -p 80 janvanoorschot.nl
traceroute to janvanoorschot.nl (178.250.146.51), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  0.190 ms  0.140 ms  0.224 ms
 2  96.120.60.137 (96.120.60.137)  6.272 ms  6.258 ms  6.245 ms
 3  ae-312-1258-rur102.beaverton.or.bverton.comcast.net (68.87.217.41)  6.742 ms  6.728 ms  6.667 ms
 4  96.216.60.245 (96.216.60.245)  6.085 ms  6.071 ms  6.568 ms
 5  ae-69-ar01.troutdale.or.bverton.comcast.net (68.85.243.197)  15.439 ms  12.863 ms  12.850 ms
 6  be-36241-cs04.seattle.wa.ibone.comcast.net (68.86.93.61)  16.373 ms be-36221-cs02.seattle.wa.ibone.comcast.net (68.86.93.53)  15.648 ms be-36241-cs04.seattle.wa.ibone.comcast.net (68.86.93.61)  15.626 ms
 7  be-2113-pe13.seattle.wa.ibone.comcast.net (96.110.44.82)  16.516 ms be-2213-pe13.seattle.wa.ibone.comcast.net (96.110.44.86)  12.743 ms be-2113-pe13.seattle.wa.ibone.comcast.net (96.110.44.82)  12.637 ms
 8  ae-9.a02.sttlwa01.us.bb.gin.ntt.net (129.250.66.105)  12.379 ms  12.333 ms  12.304 ms
 9  ae-2.r24.sttlwa01.us.bb.gin.ntt.net (129.250.2.72)  15.667 ms  15.641 ms *
10  ae-11.r20.nwrknj03.us.bb.gin.ntt.net (129.250.6.176)  76.058 ms  76.043 ms  76.029 ms
11  ae-9.r20.londen12.uk.bb.gin.ntt.net (129.250.6.146)  147.383 ms  147.368 ms  147.871 ms
12  ae-11.r20.amstnl07.nl.bb.gin.ntt.net (129.250.5.2)  154.671 ms  146.154 ms  146.773 ms
13  ae-0.a00.amstnl07.nl.bb.gin.ntt.net (129.250.7.65)  159.230 ms  159.198 ms  157.998 ms
14  xe-1-5-1-0-4070.a00.amstnl07.nl.ce.gin.ntt.net (81.20.69.222)  162.916 ms  156.622 ms  161.692 ms
15  rc03-po4.core.as41887.net (94.228.128.125)  153.436 ms  147.642 ms  146.584 ms
16  rc02-xe-0-0-3.bbd.as41887.net (94.228.128.58)  149.174 ms  153.629 ms  145.518 ms
17  dev.robomindacademy.com (178.250.146.51)  150.546 ms  150.441 ms  144.811 ms
18  dev.robomindacademy.com (178.250.146.51)  146.009 ms  149.976 ms  154.788 ms
1 Like

Yes, because it is curl that uses user-agent with -A. nmap -A is something else

I don't see any evidence of Palo Alto firewall but it does look like a firewall blocking the IP address of the Let's Encrypt validation server(s).

The Let's Debug initial test works fine (gets a 404) but the test to Let's Encrypt staging system times out.

I can also reach that domain from various IP's around the globe so looks like a regular IP based block to me

3 Likes

Sorry; my bad! :frowning:

2 Likes

Also interesting
image

1 Like

I bow low. This is my office firewall, a Turris router. It installed (and activated!) a 'Advanced Security & Analytics - Turris Sentinel' package that contains a Dynamic Firewall. I disabled/uninstalled this package and now i can install certificates.
So Mike seems to be right, it was a firewall block. I have no idea why that Turris Sentinel thing blocks the Let's Encrypt staging system.

Thanks for your support!! ... this was one i could not have done on my own.

Jan

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.