Renewal stopped working

Help
1

I've searched through posts for two days now and tried so many variations on the fixes that others have used but no joy. This site has been successfully auto-renewing for four years and suddenly it complains about reaching the challenge token files. But when they get created I'm able to access them fine in my browser (from outside the network so it's not a firewall issue).

I've disabled ufw on the server just to eliminate any chance of it interfering. I've tried both with and without our standard http -> https redirect in place in the vhost config.

The cert expires in just over two weeks and I'm at a loss. Any help is greatly appreciated. I can post the contents of the log file if that's helpful but I didn't want to overload this with unnecessary detail until it's actually helpful. Questionnaire info is below.

Thanks.

My domain is:
highbridgefilmfestival.org (plus 5 aliases, all of which have certificates)

I ran this command to get my certificate:
certbot renew
certbot renew --webroot --webroot-path /web/highbridge

My web server is (include version):
Apache/2.4.29

The operating system my web server runs on is (include version):
Ubuntu 18.04.6 LTS

My hosting provider, if applicable, is:
N/A - local server

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version if you're using Certbot):
certbot 3.0.1

2

For some reason I cannot fathom, you've left out probably the most important question and answer of the entire questionnaire:

It produced this output:

That said, running your domain through Let's Debug (Let's Debug) shows:

highbridgefilmfestival.org has an A (IPv4) record (148.59.183.132) but a request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address.
A timeout was experienced while communicating with highbridgefilmfestival.org/148.59.183.132: Get "http://highbridgefilmfestival.org/.well-known/acme-challenge/letsdebug-test": context deadline exceeded

And Let's Encrypt shows a similar error:

148.59.183.132: Fetching http://highbridgefilmfestival.org/.well-known/acme-challenge/xXMTMXk3xcrkEZn-CwIIc12V_bNkep46wFo8CAe4py0: Timeout after connect (your server may be slow or overloaded)

3

Do you have a Palo Alto brand firewall by any chance? Because I see a problem similar to what we saw with that brand last year.

Or, just some kind of firewall that inspects user-agent value?

An HTTP request with the default curl user-agent gets instantly redirected to HTTPS

curl -i -m4 http://highbridgefilmfestival.org/.well-known/acme-challenge/Test404
HTTP/1.1 301 Moved Permanently
Server: Apache/2.4.29 (Ubuntu)
Location: https://highbridgefilmfestival.org/.well-known/acme-challenge/Test404

But, a user-agent like Let's Encrypt uses gets a timeout

curl -i -m7 http://highbridgefilmfestival.org/.well-known/acme-challenge/Test404 -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (28) Operation timed out after 7001 milliseconds with 0 bytes received

I can repeat this consistently. You can see a similar result using https://letsdebug.net

5 Likes
4

I think it is a PA but I'll check with the network team to be sure. But it's been in place for quite a while and renewals have been working fine with it. Not that that discounts the potential of it being an issue. Just a curious data point. I'll verify it with them and see if they're doing any filtering on user agent.

1 Like
5

The PA issues we saw (much) earlier involved PA changing a default for a config setting. It went from allowing inbound HTTP Challenge requests to denying them (ouch).

Looking back I see it started in early 2022 but had people reporting problems as late as earlier this year. This is just one of many examples we saw:

3 Likes
6

The network team confirmed that it is a PA. And then they also told me that they've implemented a new firewall since the last successful renewal in September. This little detail seemed not important to them on Wednesday when I asked what might have changed recently. Funny that... (smh) It was supposed to be like for like as far as policies go but obviously that's not the case. Or at least it would appear so.

The guy who's set it all up was already gone for the day so we're gonna pick it back up on Monday. I really appreciate the push in the right direction.

6 Likes
7

That was exactly the issue and I really appreciate your assistance! Once they allowed acme-protocol it worked fine.

4 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.