Some challenges have failed due to geographic blocking

[bbarto99]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
dtcflightservices.com, www.dtcflightservices.com

I ran this command:
sudo certbot certonly --dry-run -v --debug-challenges

It produced this output:
Waiting for verification...
Challenge failed for domain dtcflightservices.com
Challenge failed for domain www.dtcflightservices.com
http-01 challenge for dtcflightservices.com
http-01 challenge for www.dtcflightservices.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: dtcflightservices.com
Type: connection
Detail: During secondary validation: 96.80.133.193: Fetching http://dtcflightservices.com/.well-known/acme-challenge/aiuo36bwLqeICncBbr2oLAmHv5RHj1XjLwGnGnTYZW4: Timeout during connect (likely firewall problem)

Domain: www.dtcflightservices.com
Type: connection
Detail: During secondary validation: 96.80.133.193: Fetching http://www.dtcflightservices.com/.well-known/acme-challenge/r3Dshv2neOP0YnjiN8-qnJWYIrzwywfVKrpeKIPaRnk: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
Apache/2.4.58 (Unix) (built Mar 6 2024)

The operating system my web server runs on is (include version):
Red Hat Enterprise Linux 9.2 (Linux 5.14.0-284.30.1.el9_2.x86_64)

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.10.0

I reviewed the Apache error logs and our firewall logs. The Apache error logs show multiple validation locations correctly being served the challenge response file with a 200 server result. However, the firewall logs do show that some connections from the Internet are being dropped. This is part of our firewall's security stance as we don't accept connections from many countries outside of North America and parts of Europe. Our experience has been that many, many of the network probing and phishing attempts come from these countries. Since we do not have a customer presence in those areas, we choose to use the firewall's geographic block capabilities to limit our exposure.

I understand that Let's Encrypt is now using multiple IP addresses from different hosts to do the validation. However, is there some way to provide at least the countries from which the validations originate (since LE does not provide lists of the actual IP addresses)?

[JamesLE]

@bbarto99

However, is there some way to provide at least the countries from which the validations originate (since LE does not provide lists of the actual IP addresses)?

No, they may change without notice.

[jvanasco]

"Blacklist" is an outdated legacy term with problematic connotations. The industry has consciously moved to abandon it in favor of "blocklist" or "denylist".

To handle geoip issues with challenges, people have found the following techniques useful:

1- Use a firewall that does not block on the /.well-known/acme-challenge directory
2- Use a firewall or CDN that implements redirects from /.well-known/acme-challenge to an unrestricted server
3- Use a DNS-01 challenge, with challenges delegated to an unrestricted public server - such as a commercial third party server

[bbarto99]

Thanks for your reply, and for the reminder over outdated/problematic terminology.

The difficulty with options 1 & 2 are that the firewall does geoip blocking upon receiving the connection open request; it does not have a way to temporarily accept a connection in order to defer a decision until some message traffic is received to evaluate.

(Nor do I have the liberty to just swap out very expensive firewalls! It would be less expensive to simply not use Let's Encrypt and purchase a certificate the old-fashioned way.)

I will look into how to use a DNS-01 challenge.

[jvanasco]

@bbarto99

I will look into how to use a DNS-01 challenge.

If you can operate a system outside of the firewall, I suggest using GitHub - joohoi/acme-dns: Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.

Otherwise, most people will use a 3rd party dns system that offers an API - like cloudflare - to handle this.

On your regular DNS servers, you can delegate the _acme-challenge record to a 3rd party server via CNAME. Then you only have that 3rd party system respond to the acme_challenges. This essentially firewalls your regular DNS from the server/records used in the ACME verification process, allowing you to not worry about the ramifications of compromised credentials. There is information on all this in the acme-dns docs.

[mcpherrinm]

Right now all our connections will originate from the US, Sweden, and Singapore. We will likely add more countries in the future so relying on that list isn’t advisable, but it’s not a secret.