Accidentally let certs expire and now I can't renew my certs

Help
1

My domain is:
mail.redfish-solutions.com

I ran this command:
letsencrypt --apache renew

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/mail.redfish-solutions.com.conf

Renewing an existing certificate for mail.redfish-solutions.com and 3 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: ftp.redfish-solutions.com
Type: connection
Detail: 45.33.216.244: Fetching http://ftp.redfish-solutions.com/.well-known/acme-challenge/409WWP6pDiZiir12skrhjkzc_pHVtqGKgr1teeJBX8w: Timeout during connect (likely firewall problem)

Domain: mail.redfish-solutions.com
Type: connection
Detail: 45.33.216.244: Fetching http://mail.redfish-solutions.com/.well-known/acme-challenge/phAwLJhYB1NNIs14aEwQPOVsHKxERaSIJQxCJllGmFA: Timeout during connect (likely firewall problem)

Domain: ntp.redfish-solutions.com
Type: connection
Detail: 45.33.216.244: Fetching http://ntp.redfish-solutions.com/.well-known/acme-challenge/udQEIP6elV25yXsyRPpYVSz6S3Y4JkjMZVnjjT0N08I: Timeout during connect (likely firewall problem)

Domain: www.redfish-solutions.com
Type: connection
Detail: 45.33.216.244: Fetching http://www.redfish-solutions.com/.well-known/acme-challenge/iJFQpBvFIwCQushuYko7t-XdzrL5WG4RiEfaX6FOwYs: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate mail.redfish-solutions.com with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/mail.redfish-solutions.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
httpd-2.4.53-1.fc34.x86_64

The operating system my web server runs on is (include version):
Fedora 34

My hosting provider, if applicable, is:
n/a

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.22.0

2

What's curious is that I can see the authentication succeeding here:

35.91.248.101 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/DlqE_jBnvKgdJVwvXHOgMHj8SnpkpuOFQNsSpvyDD5Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/HOqPgSiCQrFUuGMWVE5k77bk9GSv-nBR0kvUR8kc4nQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/600HVVRPua1ifLFpTa4IedGNut4Qu9VoJPEgA9nLgC8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/HOqPgSiCQrFUuGMWVE5k77bk9GSv-nBR0kvUR8kc4nQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/600HVVRPua1ifLFpTa4IedGNut4Qu9VoJPEgA9nLgC8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/DlqE_jBnvKgdJVwvXHOgMHj8SnpkpuOFQNsSpvyDD5Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/IeAkYFM42_AEGXz0xGId_Qm8wkqM4_lzDJfSzYLBIM0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/600HVVRPua1ifLFpTa4IedGNut4Qu9VoJPEgA9nLgC8 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/HOqPgSiCQrFUuGMWVE5k77bk9GSv-nBR0kvUR8kc4nQ HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/IeAkYFM42_AEGXz0xGId_Qm8wkqM4_lzDJfSzYLBIM0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/DlqE_jBnvKgdJVwvXHOgMHj8SnpkpuOFQNsSpvyDD5Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:27:28 -0600] "GET /.well-known/acme-challenge/IeAkYFM42_AEGXz0xGId_Qm8wkqM4_lzDJfSzYLBIM0 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:28:18 -0600] "GET /.well-known/acme-challenge/r463M6skLe2Q8vBEmcMxnMnNdEMAtHtwjbt0vj6wm40 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:28:18 -0600] "GET /.well-known/acme-challenge/r463M6skLe2Q8vBEmcMxnMnNdEMAtHtwjbt0vj6wm40 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:28:18 -0600] "GET /.well-known/acme-challenge/Q5IOIKW_pqdCNTzY7L6g4fsRWVir9Pn-c3akm2wL3Ow HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/Q5IOIKW_pqdCNTzY7L6g4fsRWVir9Pn-c3akm2wL3Ow HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/jWtnZzJ5vCtvrHDrKLlJI2M46n4feUhDk-3RJTld51Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.248.101 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/IfRSukbcPRI_ZnF8wTV5XhX16WLMK3QORI7Xr6FVmBk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/jWtnZzJ5vCtvrHDrKLlJI2M46n4feUhDk-3RJTld51Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/Q5IOIKW_pqdCNTzY7L6g4fsRWVir9Pn-c3akm2wL3Ow HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/jWtnZzJ5vCtvrHDrKLlJI2M46n4feUhDk-3RJTld51Y HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.73.59.43 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/IfRSukbcPRI_ZnF8wTV5XhX16WLMK3QORI7Xr6FVmBk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:28:19 -0600] "GET /.well-known/acme-challenge/r463M6skLe2Q8vBEmcMxnMnNdEMAtHtwjbt0vj6wm40 HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.22.130.126 - - [04/Sep/2022:10:28:20 -0600] "GET /.well-known/acme-challenge/IfRSukbcPRI_ZnF8wTV5XhX16WLMK3QORI7Xr6FVmBk HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3

Those are all AWS IP addresses, which are the secondary validation locations. It seems the primary validation location is getting a time out. Currently (earlier this afternoon) the requests from the primary validation location were originating from the IP address range 23.178.112.0/24. Please make sure you're not blocking access from that range.

2 Likes
4

I think my geoip database was out-of-date, because it was attributing that CIDR to JP which we block due to lax AUP enforcement.

If I temporarily allow JP in my country-code list, then it succeeds.

Thanks!

2 Likes
5

Note that geoip blocking can also lead to validation errors if Let's Encrypt decides to add or change the secondary validation locations to countries you're blocking.

3 Likes
6

Note that geoip blocking can also lead to validation errors if Let's Encrypt decides to add or change the secondary validation locations to countries you're blocking.

Yeah, it's not ideal. But those countries are free to pass better cyber security enforcement laws. Hopefully the EFF picks judiciously...

7

EFF is the organisation (currently) responsible for the ACME client Certbot. However, Let's Encrypt is ran by the organisation "ISRG". While EFF is a founding sponsor of ISRG and a sponsor of Let's Encrypt, they are different organisations :slight_smile:

4 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.