Sudden renew failures but not firewall block

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: www.hanscees.com

I ran this command: certbot renew --cert-name www.hanscees.com --force-renewal

It produced this output:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: www.hanscees.com
Type: connection
Detail: During secondary validation: 213.233.217.138: Fetching http://www.hanscees.com/.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U: Connection refused

My web server is (include version):
nginx version: nginx/1.24.0
built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r4)
built with OpenSSL 3.0.7 1 Nov 2022 (running with OpenSSL 3.0.12 24 Oct 2023)

The operating system my web server runs on is (include version): docker on alpine3.19

My hosting provider, if applicable, is: home
I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.10.0

Hi,
I have been running certbot for a long time and it suddenly started to fail. Either my nginx config has an error (but I checked thoroughy), or your service has changed something perhaps.

I have red your notice of firewall blocking, but when the renew failes I see the check in my access logs. So it looks like another problem.
51.21.3.3 - - [07/Jun/2024:09:08:45 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/I6Jjk4LOnp8HGG61OOdWwFgSh4mEBjKsYXA-KVsvDgk HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/I6Jjk4LOnp8HGG61OOdWwFgSh4mEBjKsYXA-KVsvDgk" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.108 - - [07/Jun/2024:09:53:20 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.237.223.61 - - [07/Jun/2024:09:53:20 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.139.226.98 - - [07/Jun/2024:09:53:20 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.108 - - [07/Jun/2024:09:53:20 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.237.223.61 - - [07/Jun/2024:09:53:20 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.139.226.98 - - [07/Jun/2024:09:53:21 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/h5oiCo_C8r_f7-lX_ORBUCgXHgYIyIsA5eoDImmCw6U" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

1 Like

Welcome @hanscees

There are only three successful responses to the challenge. You can see there are three that result in a 301 redirect. And then three with the 200. The first one was for a different challenge as it has a different URI token value.

I looked up the countries for those IP and they were two in the US and one in Singapore. So you are missing one other from the US and one from Sweden. Again these can change at anytime but that's how it is now

You should be seeing at least 4 and usually 5 successful 200 status responses for each successful challenge. That could change in the future but that's how it is right now.

Something is preventing some of the requests from reaching you. You are either blocking specific IP or IP ranges. Or perhaps some kind of geographic area

4 Likes

Please don't use this option if you don't know what it means (as a general rule: never use options you don't know the effect of). Especially, it does not magically force the ACME server to make authorizations valid.

1 Like

Hi @hanscees,

Using the online tool Let's Debug yields these results Connection refused
https://letsdebug.net/www.hanscees.com/2012186?debug=y

IssueFromLetsEncrypt
ERROR
A test authorization for www.hanscees.com to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
213.233.217.138: Fetching http://www.hanscees.com/.well-known/acme-challenge/s3ozNhjEcqfe7SnOe6i1t8DNUs8hW1Y2NmlajSJaYPg: Connection refused

Yet from my Oregon USA location using curl this is what I see.
The response for the HTTP request:

$ curl -Ii http://www.hanscees.com/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 07 Jun 2024 16:17:56 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://www.hanscees.com/.well-known/acme-challenge/sometestfile

The redirection from HTTP to HTTPS response:

$ curl -Ii https://www.hanscees.com/.well-known/acme-challenge/sometestfile
HTTP/2 404
server: nginx
date: Fri, 07 Jun 2024 16:18:08 GMT
content-type: text/html
content-length: 146
1 Like

assumptions. I am using this to renew a domain that is not yet expired because I know the configuration in nginx is good.

First of all, yes I have several layers of security. One of them is geoblocking, but thats not on port 80 nor 443. The other is fail2ban that kicks you off if you try a few times too many. So the connection can be closed for several reasons.

Two questions:

  1. Since geoblocking is essential to me, is there a list of ip's somewhere I can allowlist?
  2. Isn't three successful connections enough to ensure dns attacks security? Why does the service fail with three? Isnt that a bit over-zealous, since it results in users being locked out: which is also a security problem

To me, this statement doesn't make much sense. Why try to renew a certificate when the certificate isn't due for renewal? And what does that have to do with the nginx configuration?

For testing purposes you should use --dry-run.

2 Likes

Hi @hanscees,

Please check out

3 Likes

well, this is strange. It now all works fine again. I was wrong previously in that I use geoblocking: thats only for email, not for http(s). So that could not have been a problem.

I think what happened is that in one domain the nginx config was missing port 80 redirects. And perhaps while troubleshooting I hit some limits on the service and broke some stuff trying to find out and perhaps fail2ban also started banning stuff.

Anyway, the renewals work now and I get four challenges per domain renewal as you can see below. Sorry to waste your time, and thanks for the great letsencrypt services.

13.49.68.129 - - [07/Jun/2024:20:08:58 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
13.49.68.129 - - [07/Jun/2024:20:08:58 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 200 87 "http://ag.hanscees.com/.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.106 - - [07/Jun/2024:20:08:58 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.237.223.61 - - [07/Jun/2024:20:08:59 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
54.169.225.5 - - [07/Jun/2024:20:08:59 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.237.223.61 - - [07/Jun/2024:20:08:59 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 200 87 "http://ag.hanscees.com/.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.106 - - [07/Jun/2024:20:08:59 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 200 87 "http://ag.hanscees.com/.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
54.169.225.5 - - [07/Jun/2024:20:09:00 +0000] "ag.hanscees.com" "GET /.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k HTTP/1.1" 200 87 "http://ag.hanscees.com/.well-known/acme-challenge/HM9kVJOaMwY3cNWGHviaq0wrBGJbSVtwGM3cxU-it7k" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

16.171.152.223 - - [07/Jun/2024:20:08:33 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.204 - - [07/Jun/2024:20:08:33 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
16.171.152.223 - - [07/Jun/2024:20:08:33 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.237.223.61 - - [07/Jun/2024:20:08:34 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
54.179.253.134 - - [07/Jun/2024:20:08:34 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
23.178.112.204 - - [07/Jun/2024:20:08:34 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
18.237.223.61 - - [07/Jun/2024:20:08:35 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
54.179.253.134 - - [07/Jun/2024:20:08:35 +0000] "www.hanscees.com" "GET /.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc HTTP/1.1" 200 87 "http://www.hanscees.com/.well-known/acme-challenge/_CCmrlJ3S2_Jft61072ojG1cXg46Mlw7GpyrF8yHuuc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

1 Like

What IP addresses does Let’s Encrypt use to validate my web server?
Let’s Encrypt does not publish a list of IP addresses we use to validate,
and these IP addresses may change at any time.

Let's Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

Please read these:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.