Specify source country for http-01 challenge

I understand the reasons that LetsEncrypt won't publish the IP addresses for the challenge servers. I also know about DNS-01 (which will not work for us since we need to manually update DNS with our hosting company).

My suggestion is to implement a directive in the renewal configuration file to specify which country the challenge comes from. There could be two forms for this:

  1. challenge_allow = code,code,code
    The challenge only comes from servers that are located in countries with these country codes.
  2. challenge_block = code,code,code
    The challenge will never come from a server in a country with the listed country codes.

Examples:
country_allow = US,CA,UK
http challenges would only come from servers in the United States, Canada, or the United Kingdom.

country_block = US,CA,UK
http challenge requests would not come from servers in the United States, Canada, or the United Kingdom.

Implementation:
When a new cert is requested, it goes through the normal process, but the process that triggers the challenge only goes to a server that is in an allowed (or not in the blocked) country.

This does not fully solve some problems people have with the challenge servers IP addresses from being on block lists, but it does solve problems for those who use GeoIP fencing on their webservers.

This would defeat the purpose of multi-perspective validation. The idea is that, by performing validation from many unpredictable locations around the world, we can reduce the likelihood that a local or targeted BGP hijack attack affects all of our validation perspectives, and thereby prevent the attack from being effective.

If a malicious client could control what perspectives we perform validation from, then they could control what perspectives they need to attack.

11 Likes

For us, the "many unpredictable locations around the world" will mostly get blocked since all of our clients (and therefore, the only traffic we want to allow) is from the US. Everything else is dropped by default.

If it were "many unpredictable locations around the US", we wouldn't see the problem.
Alternatively, if there were multiple challenges all at the same time from many different servers (and any one of them succeeded), we also wouldn't see the problem (as long as at least one was in the US).

The current behavior of the challenge seems to be, "ask once, fail if blocked." (based on the logs). Given that position and the "fail 5 times in an hour and you're in time-out" makes it more difficult to have an automated certificate renewal that "just works". If you stick with this general model, I'd suggest something like "have 10 sites challenge, ignore non-connections (firewalls), accept on one success, fail on no successes."

If a malicious client could control what perspectives we perform validation from, then they could control what perspectives they need to attack.

Correct me if I'm wrong, but the argument is: if someone tells certbot to get the cert from some particular country, that's going to provide some insight into what IP address prefixes they need to try and advertise to maliciously route traffic.
Traffic that will provide them with a certificate (and no matching key)?

I don't understand how this is a mitigation of risk considering the ease of obtaining the same information in other ways. (I'm honestly curious, feel free to DM me if you don't want it public.)

Side note: According to the Cloudflare site, keeping network routing paths as short as possible is a mitigation against BGP hijacking. By that reasoning, the challenges should come from sources closer to the requestor.

Thanks!

@kcolagio that is a choice your organization made. But there is more than one way to skin a cat.
I do not believe the choice your organization made should be a constraint on all others.

There are other Free ACME CAs

For further requirement for CAs please see CA/Browser Forum Baseline Requirement .

And Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

4 Likes

There will be a hard requirement (from 2026 oct iirc) for CA to validate domain from at least two different RIR zone (slight different but mostly per continent) so blocking outside USA won't work

8 Likes

I think you may be missing the point of what a CA does: Its entire purpose is to ensure that the controller of a domain name at the time of getting a certificate is the same as the controller of the server that a user is trying to connect to. In order to ensure that the entity trying to get a certificate in fact controls the name as seen by everyone on the Internet, they need to check how it looks from everywhere on the Internet (or at least as close as they reasonably can).

If you need to do some sort of geo-blocking, some common approaches are to try to exempt the requests for /.well-known/acme-challenge from whatever firewall is being used, or hook into the validation process to script opening up the firewall before attempting validation and then close it again afterward. And also keep in mind that this only needs to be done for port 80, you can keep all your "real" application geo-blocked on port 443 the entire time.

4 Likes

Although you mentioned DNS-01 won't work for you, this might be helpful to other readers: one way to use DNS-01, even if you can't programatically update records on your main DNS host, is to delegate the _acme-challenge subdomain(s) (using NS records) to another host or service that does offer an API, for example: GitHub - joohoi/acme-dns: Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.

7 Likes

As an organization that needs to meet certain security compliance standards, we don't entirely have a choice in how we block traffic.

Thanks for the extra links though. Much appreciated!

@kcolagio here is a list of DNS providers who easily integrate with Let's Encrypt DNS validation

2 Likes

You only have to allow worldwide access to limited resources, ie ${fqdn}/.well-known/acme-challenge via http or _acme-challenge.${fqdn} on dns. If you have to block http, I hope you won't have to block dns.

5 Likes

I cant speak for why ISRG requires things, but my opinion is:

Responding in reverse, as this order may make more sense:

The multiple validation points mean there are multiple networking attempts, against which all must succeed.

Assume your site is A1, and we are trying to validate from B1 in the USA. If there is only one validation attempt from B1, a successful BGP attack can happen anywhere between A1 and B1. If there are multiple validation attempts, then each path needs to be similarly compromised – somewhere between A1<->B2 EU, A1<->B3 AU, A1<->B4, etc.

The multiple vantage points directly translates to multiple networks requiring compromise in order for a certificate to issue. It significantly increases the complexity of the attack.

Under that framework, someone could use a DDOS attack to offline 9 nodes and use a BGP attack on only 1 node - effectively making it easier to obtain a malicious cert.

There are certainly other reasons for the above, these are just the most topical examples I thought of.

In terms of security considerations, I hope you can recognize your trouble with LetsEncrypt is that it is requiring more security than you believe is necessary. While part of this is from CA/B forum mandates, a larger part of this is simply driven by the organization's real concern to defend against common attack vectors. Your organization decided to limit access as a security measure, which works in some ways, but not in others. That has made your organization's goals, and the available selection of tools, particularly limited and approaching incompatibility with global solutions such as LetsEncrypt.

Perhaps it is better stated like this: LetsEncrypt helps secure websites by requiring global validation to ensure a wide array of routing paths are not compromised. By deciding to geofence your network, while your organization limited the number of routing paths that can be compromised, it also restricted itself out from utilizing this mechanism for protection.

7 Likes

The other thing to keep in mind is that even if you're intending your domain name to only be used from within one specific country, the certificates that Let's Encrypt (or any other CA) creates are valid for use globally. So they need to protect against someone from some other country trying to intercept traffic for your domain name, even if that's not a threat that your organization is especially worried about.

5 Likes

This is a great tip. You can also do this with a CNAME in some cases, right?

Can we maybe make a new page for in Documentation - Let's Encrypt for some form of "I can't allow incoming connections from the whole world"? Maybe especially with the recent changes, some form of this seems to be a pretty frequently-asked question. I'll make a little draft.

Edit: Here's a very basic draft on this issue.

2 Likes

Yes, in all cases. That's the premise of the acme-dns usage pattern, but it can be used for any commercial vendor or self-managed system.

The Public DNS system (main DNS) hosts an _acme-challenge record that CNAMEs onto some secondary DNS system. LetsEncrypt follows the CNAME.

The secondary DNS system hosts the actual challenge record, and is manipulated by Certbot via an API. The secondary system is limited to only handling acme-challenges, so the security risk of the credentials being on the server is negligible.

This is why I try to always say something like "there are other examples why", because someone here will always chime in with another great and extremely relevant example.

3 Likes