Have renewed certificate fine for years but now i can't renew it

And now for the smoking gun; potentially a firewall issue.

$ curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer
$ curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 30 Jan 2023 19:31:22 GMT
Content-Type: text/html
Content-Length: 354
Last-Modified: Mon, 30 Jan 2023 14:51:12 GMT
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: "63d7d960-162"
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains
Accept-Ranges: bytes

1 Like

Yes, very sus output.

Is there a Palo Alto firewall in use?

4 Likes

Guys (@Bruce5051 + rg305), first of all, thanks !!

I got the nginx response 403 (forbidden) in non-existing server struture. Don't know, if that different of 404 error implies certbot to not working.

I have doubts that TI people has made any change in firewall... My big doubt is why that stops working suddenly. It is my first problem of renewall since 2019.

curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/

HTTP/1.1 403 Forbidden
Server: nginx
Date: Mon, 30 Jan 2023 20:31:58 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
Strict-Transport-Security: max-age=31536000; includeSubDomains

Below it's ok cause there is a temporary index.html... Originally, server redirects 80 to 443... it is not doing that cause i am trying correct certbot renewal.

curl -Ii http://w3.dmat.ufrr.br/

HTTP/1.1 200 OK
Server: nginx
Date: Mon, 30 Jan 2023 20:32:46 GMT
Content-Type: text/html
Content-Length: 354
Last-Modified: Mon, 30 Jan 2023 14:51:12 GMT
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: "63d7d960-162"
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=31536000; includeSubDomains
Accept-Ranges: bytes

Thanks again !!
3 Likes

The difference between HTTP status codes 403 and 404.

2 Likes

I guess the server is responding HTTP 404 correctly. But Lets Debug is still responding not ok.

https://letsdebug.net/w3.dmat.ufrr.br/1355928

curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challege/kfjsadfjsdljf

HTTP/1.1 404 Not Found
Server: nginx
Date: Mon, 30 Jan 2023 21:15:27 GMT
Content-Type: text/html
Content-Length: 313
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: "63d83359-139"
Strict-Transport-Security: max-age=31536000; includeSubDomains

1 Like

About Lets Debug answer... the server has a valid IPv4 (200.129.159.22) physically set up on the interface, not NAT or anything else.

ifconfig

em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=481009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
	ether c6:c6:c8:59:dd:14
	inet 200.129.159.22 netmask 0xfffffff0 broadcast 200.129.159.31
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
1 Like

It might be working for you (from the same network perhaps?), but from (a large part of) the public internet your website seems to be down. Connecting to port 80 from my point of view results in a timeout.

4 Likes

nmap to find what is open

$ nmap -Pn w3.dmat.ufrr.br
Starting Nmap 7.80 ( https://nmap.org ) at 2023-01-30 21:35 UTC
Nmap scan report for w3.dmat.ufrr.br (200.129.159.22)
Host is up (0.21s latency).
Not shown: 939 filtered ports, 59 closed ports
PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 9.38 seconds

Why does this fail?

$ curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer

And this passes, they are basically the same the failing one has added
-A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
to emulate Let's Encrypt.

$ curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Server: nginx
Date: Mon, 30 Jan 2023 21:34:09 GMT
Content-Type: text/html
Content-Length: 313
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: "63d83359-139"
Strict-Transport-Security: max-age=31536000; includeSubDomains

2 Likes

Hi Osiris,

Not at same network... at same city, yes.... and @Bruce5051 showed me the inconsistency of server access across the world via check-host.net. That's Ok.

My first goal is correct the server as perfect as possible to not hold it responsible for any impediment to certificate renewal, i just may solve the machine problem. As i told before, it is the first renewal problem in years.

And i guess that kind of network inconsistency is not showing up only now... and it still be happen. And i have no way to solve it.

Another question is, would the dns challenge be more effective at that scenario ??

To imagine that server running without crypto, let me shaking.

Thanks a lot guys !!!

4 Likes

Is there a Palo Alto firewall between your server and the Internet?

3 Likes

Also @jlgm Is there any GeoLocation blocking?
Potentially new GeoLocation blocking that you were not made aware of?

2 Likes

@jlgm You are almost certainly affected by a Palo Alto brand firewall. You have the same symptoms as we often saw earlier in 2022. See (this link) for more info

You should talk to your network admins and have them change the Application Rule for "ACME protocol".

The tests described by my link above for your domain are this:

(test gets expected 404 with test file)
curl -I http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Server: nginx

(test fails when using a user-agent the same as Let's Encrypt servers)
curl -I http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer
4 Likes

Here's the semi-official thread describing the problem:

5 Likes

And still seeing those same results:

$ curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer
$ curl -Ii http://w3.dmat.ufrr.br/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Server: nginx
Date: Tue, 31 Jan 2023 17:34:50 GMT
Content-Type: text/html
Content-Length: 313
Connection: keep-alive
Keep-Alive: timeout=65
Vary: Accept-Encoding
ETag: "63d83359-139"
Strict-Transport-Security: max-age=31536000; includeSubDomains

3 Likes

Hi guys !!

Renewal done Ok !!

The delay was due to feedback of firewall's admin. Don't know if there is a palo alto fw, but it was a firewall problem indeed.

Thanks a lot for your help.

JL

PS: BTW, change challenge to dns has some improvement, is it an easy procedure ?? TKS

6 Likes

I am still without that answer to you about GeoLocation issue, @Bruce5051... Sorry !

3 Likes

Give that

I don't think the GeoLocation is presently of significance. Thanks! :slight_smile:

3 Likes

It can be easy.
But it can also be impossible [to automate].
It depends mostly on the DSP [DNS Service Provider] in use and then also your willingness to meet whatever requirements remain unmet.

4 Likes

Check here for DNS providers who easily integrate with Let's Encrypt DNS validation

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.