Renewal says fetch timed out, but logs say fetch succeeds

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: sandbox.mc.edu (also sandbox-v01.mc.edu) Both fail in apparently the same way.

I ran this command: certbot -v renew

It produced this output:

[root@sandbox bennet]# date ; certbot -v renew
Mon Apr 15 03:40:42 PM CDT 2024
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/sandbox-v01.mc.edu.conf

Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for sandbox-v01.mc.edu
Performing the following challenges:
http-01 challenge for sandbox-v01.mc.edu
Waiting for verification...
Challenge failed for domain sandbox-v01.mc.edu
http-01 challenge for sandbox-v01.mc.edu

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: sandbox-v01.mc.edu
Type: connection
Detail: During secondary validation: 167.160.210.32: Fetching http://sandbox-v01.mc.edu/.well-known/acme-challenge/XIliGDiaLLaiOGiRbWQjxS3yMShwSH9dmkpwdDoPagw: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate sandbox-v01.mc.edu with error: Some challenges have failed.

Processing /etc/letsencrypt/renewal/sandbox.mc.edu.conf

Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for sandbox.mc.edu
Performing the following challenges:
http-01 challenge for sandbox.mc.edu
Waiting for verification...
Challenge failed for domain sandbox.mc.edu
http-01 challenge for sandbox.mc.edu

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: sandbox.mc.edu
Type: connection
Detail: During secondary validation: 167.160.210.32: Fetching http://sandbox.mc.edu/.well-known/acme-challenge/cKmMjOSnsuuznbiz0MJNYOZbY4ewEKea_Y-eXEgC-o8: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate sandbox.mc.edu with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/sandbox-v01.mc.edu/fullchain.pem (failure)
/etc/letsencrypt/live/sandbox.mc.edu/fullchain.pem (failure)

2 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
httpd-2.4.58-1.fc38.x86_64 (Fedora packaged version, as you can see)

The operating system my web server runs on is (include version):
Fedora release 38 (Thirty Eight)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.9.0

Certbot, as you can see, says it cannot connect to the web server to retrieve its test document. But the Apache log says it worked fine.

35.89.9.72 - - [15/Apr/2024:15:41:06 -0500] "GET /.well-known/acme-challenge/cKmMjOSnsuuznbiz0MJNYOZbY4ewEKea_Y-eXEgC-o8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Other fetches also show success.

And Wireshark shows the full transaction from syn to fin|ack, fin.

2

Let's Encrypt validates from multiple perspectives around the world.

We've recently added two new perspectives in Sweden and Singapore. Make sure you don't have any geographic filters blocking those countries.

3 Likes
3

You should be seeing four or five successful requests in your log for a cert to be issued. There was a recent change to remote validation sites. Do you block access to various countries or IP addresses?

3 Likes
4

Well, the entries in the Apache log geolocate to the US. (As I said the supposed time-outs succeed according to apache and Wireshark logs.)

Would a single renewal involve multiple fetches from multiple addresses in different locations?

5

There's some luck involved.

6

Yes. As I noted ... you should see at least 4 and possibly 5 records in your access log for a successful validation. There are (currently) 5 requests from various global points with (currently) 4 needing to succeed to satisfy the challenge.

There are 3 Let's Encrypt validation centers in the US and 2 outside it.

3 Likes
7

Thank you.

I don't think we have any particular geoblocking. I'll see what IT
says. In any case, the Apache logs and Wireshark traces show successful
connections and retrievals.

There are two renewals (not new issues if that matters), and I show
three successful HTTP queries for each one. All three show the same
URL. Do the URLs vary through the process, or am I seeing the first
one tried three times before the CA server gives up?

The error message from certbot specifically says this URL times out.
But I see it successfully fetched in my Apache logs. What sort of
geoblocking would do that anyway? It would presumably block the
request, rather than complete the TCP session and then discard the
response.

3.16.128.60 - - [15/Apr/2024:15:41:06 -0500] "GET /.well-known/acme-challenge/cKmMjOSnsuuznbiz0MJNYOZbY4ewEKea_Y-eXEgC-o8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.209 - - [15/Apr/2024:15:41:06 -0500] "GET /.well-known/acme-challenge/cKmMjOSnsuuznbiz0MJNYOZbY4ewEKea_Y-eXEgC-o8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.89.9.72 - - [15/Apr/2024:15:41:06 -0500] "GET /.well-known/acme-challenge/cKmMjOSnsuuznbiz0MJNYOZbY4ewEKea_Y-eXEgC-o8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

All three of those source addresses geolocate in the US.

1 Like
8

It doesn't. A "renewal" is a new cert with the same domain names. It must go through the same validation. It's kind of a confusing term but is useful when discussing rate limits and other bits.

The same URL is requested once from all 5 LE validation centers. You only see 3 because 2 are being blocked somewhere.

I can see why you say that but that's not what the error means. It is describing the end result of the challenge. And, it failed as 2 of the 5 requests timed out. A timeout is the reason the challenge failed. The URL is given as further info. It is helpful in cases where people have redirected the original URL. Which is allowed with certain constraints.

Any geo block denying requests from non-USA origins :slight_smile:

That is almost certainly what is happening here.

3 Likes
9

Thank you. All those requests use the same URL then? That would make
some more sense out of it, if the requests all look the same, but some
are just not getting through.

1 Like
10

Yes, that is correct

3 Likes
11

Is there any way to get certbot to tell me the source IPs of the failing
fetches? Geolocation is not an exact science, after all.

Thanks.

According to the IT guys, our ISP blocks from just these:

Afghanistan
Belarus
Brunei
China
Iran
Iraq
Mongolia
Myanmar
North Korea
Pakistan
Russia
Syria
Tunisia
Turkmenistan
Uzbekistan
Venezuela

1 Like
12

None of those countries are currently being used.

All our requests currently come from the USA, or AWS eu-north-1 and ap-southeast-1. Since your problems are those two AWS regions, you could check Amazon's publishes IP ranges for them. Note that we use ephemeral ec2 instances which are routinely replaced, so there's no static list of IPs, and they can change every request. If you wanted to run your own EC2 instance in those regions, you could test your own connectivity.

4 Likes
13

There is one potentially relevant edge case here: LetsEncrypt caches successful validations (currently 30 days). If LetsEncrypt is able to successfully validate one of your domains (from all vantage points), it will not try to re-validate that domain if that domain is used in other orders by your account within the cache period.

This means if only one of two domains passes, you should not see any requirements or attempts to validate it on the retry attempts.

This design implementation of LetsEncrypt can often confuse and frustrate users who are troubleshooting connectivity and vantage point issues. It can make the troubleshooting changes that actually break a system appear to be fully functional.

4 Likes