Multi-Perspective Validation & Geoblocking FAQ

(This post is a wiki; other community members are welcome to edit and improve it!)

What is Multi-Perspective Validation?

Let's Encrypt needs to connect to your DNS server, and in most cases (for the HTTP-01 or TLS-ALPN-01 challenge type) also your web server, in order to validate that your client requesting a certificate actually has control over the domain name it's asking for. This is what's meant by "Validation"; see the How It Works documentation page for more details. What "Multi-Perspective" means is that Let's Encrypt checks from multiple places, to make sure that control over the domain is established the same way when looking from different parts of the Internet. (Yes, the Internet does look different depending on where one is looking from.)

For more details, refer to this blog post that Let's Encrypt made in 2020 when they first implemented it:

Why does Let's Encrypt do Multi-Perspective Validation?

Let's Encrypt (like any other public Certificate Authority) has to ensure that your domain is actually controlled by you and not an attacker so that it can issue you a certificate. But if an attacker can intercept traffic intended for your DNS server or web server IP, then the attacker might be able to get a certificate for your domain name. Checking from multiple places makes this much harder for an attacker, because rather than trying to intercept traffic in just one place on the Internet, they would need to intercept traffic in a whole bunch of places.

Are these kinds of attacks against CAs just theoretical, or have they actually happened?

They're certainly not common, but yes they have happened. A recent high-profile one was in August 2022, where an attacker did a BGP attack (getting traffic for an IP range to route to their servers instead of the domain owner's servers) against a cryptocurrency service, and once traffic was routed to them they managed to get a valid certificate (not from Let's Encrypt, but a different CA). (You can see Coinbase's writeup on this "Celer Bridge incident" if you want the full story of that attack, but the BGP attack and getting a fraudulent certificate are just one piece of that story.)

Is Let's Encrypt the only CA doing Multi-Perspective Validation?

Let's Encrypt is leading the industry, as they often do. It is also implemented by other CAs such as Google Trust Services which calls it "Multi-perspective Issuance Corroboration". Even more CAs are currently in the process of working to add Multi-Perspective Validation. The CA/B Forum's Validation working-group largely supports incorporating this into the CA/B Forum Baseline Requirements — which all CAs must follow — and support from key forum members suggests this will become an official requirement in the near future.

Where can I read more technical details on how Multi-Perspective Validation helps?

If you look at the CA/B Forum proposal for all CAs to validate from multiple perspectives, there's an "Additional Resources" section with links to lots more information, including academic papers on the subject.

Where does Let's Encrypt currently validate from?

Well, it keeps changing. They started with 4 perspectives in 2020, went down to 3 briefly at the end of 2022 while they were regrouping and figuring out exactly where they wanted to validate from, and went back up to 5 at the end of March 2024. They are now once again checking from multiple countries, and expect to keep adding more.

The specific places they currently validate from isn't exactly a state secret, and they'll generally let you know if you ask nicely. But they expect to keep changing these places over time, as they work out the best way to protect from attacks, so any firewall rules based on specific countries won't help over time.

But what countries do I allow through my firewall in order to get a certificate?

Because the validation process is from multiple places, which can and will continue to change, we don't recommend geoblocking the requests that are used in order to validate your domain name and allow you to get a certificate.

But opening my firewall up seems terrible for security! What's the minimum I need to allow?

You can certainly do lots of blocking of traffic to your network that you don't want, but in order to get a certificate you will need to allow this traffic in order for Let's Encrypt to validate you control your domain name during the domain validation phase of the ACME protocol:

  1. For all challenge types: Worldwide access to your public authoritative DNS server
  2. For the HTTP-01 challenge (used most commonly): Worldwide access to port 80 (unencrypted http) for requests to the /.well-known/acme-challenge/ path.
  3. If using the TLS-ALPN-01 challenge (which is really rare): Worldwide access to port 443 (https).

Put another way, here's what you can block, for the most common HTTP-01 method of validation:

  1. Anything you want to block on port 443 (https).
    • Caveat: If the way your ACME client interacts with your web server sends a redirect request from http to https, then your server will need to handle that redirected request on https port 443 in order to get a certificate. This isn't the common way that ACME clients work, however.
  2. Any traffic on port 80 (http) which isn't for the /.well-known/acme-challenge/ path.
    • It's very common for port 80 to only handle redirecting to https, so allowing all traffic on port 80 is usually not much of a security risk.
  3. Anything you want on any port — if it's not a time that you're requesting a certificate. That is, some people use advanced setups having a script called by their ACME client (such as certbot's --pre-hook and --post-hook) that opens up the firewall for the duration of the challenge, and then closes it back up again after. The ACME Protocol only requires ephemeral HTTP access to your server during the validation phase of a Certificate order, you are free to block access outside of this short window which lasts about 5–10 seconds. The need for this kind of setup isn't very common.

But I only ever have users from my own country, why do I need to open something up to the world?

Even if you're only expecting traffic from one place, the certificates that Let's Encrypt issues are valid worldwide. So even if it's not an attack that you're particularly worried about, Let's Encrypt needs to validate that in some other country, there isn't some attacker trying to get a certificate for your domain. And the only way to do that is to make sure that in those other countries, you can still prove that you have control over your domain name.

What tools are out there to help check my domain's accessibility from around the world?

To be clear, these aren't provided by Let's Encrypt, but there are some web tools that community members have found useful:

I can't or won't allow traffic to my web server from everywhere. Do I have any other options?

If your web server needs to stay firewalled off from the Internet (maybe it's only intended for users in one particular organization), then the alternative is to switch to the DNS-01 challenge type. This can be a bit tricker to set up, because it involves updating records on the DNS server when getting a certificate (which is usually a different system), rather than just updating your web server. You need to have a DNS provider with an API allowing updates in order for your ACME client to do this automatically. Most popular big DNS providers do have one now, with plugins available for popular ACME clients to use them. Once set up, control over your domain can be proven with just worldwide access to your DNS server, and your web server no longer needs to accessible to the Internet at all.

One thing some need to be cautious of: Some DNS providers only offer access token credentials that allow for full access to update the entire DNS zone, which not everyone is comfortable having sitting on the web server running their ACME client. However, most big cloud providers, like AWS Route 53 and Azure DNS offer fine-grained control over DNS zone access rights, and it's becoming a more common feature. You may need to create a separate "zone" within your DNS provider's system in order to have a credential limited to just the _acme-challenge record.

If you're using a DNS provider that doesn't offer an API, you can switch to one that does, or delegate just the _acme-challenge name to another system that does. For example, the popular acme-dns tool allows you to run a special-purpose DNS server just for answering the DNS-01 challenge when getting a certificate.

Besides what Let's Encrypt is doing, how else can I protect my domain from fraudulent certificates?

Some great ways to help enhance the security of your domain and help prevent an attacker intercepting your traffic and getting a certificate include:

  1. Implementing DNSSEC.
  2. Implementing CAA records
  3. Monitoring Certificate Transparency logs using CT search tools. (This is about detection rather than prevention, but can still be helpful.)

I'm still having trouble getting a certificate, what should I do?

Don't post in this thread. Instead, create a new topic in the Help category, answer the questions provided and give as many details as you can about your system and what might be blocking the validation requests, and the friendly people in this community will see what they can do to help.

12 Likes

Google also supports multi-perspective validation which they refer to as "Multi-perspective Issuance Corroboration (MPIC)".

9 Likes

Thank you; I was pretty sure that other CAs had already started but I didn't know which ones offhand. I've added a mention and link in the post.

6 Likes

Want to merge in any of the material from this earlier draft?

(I think your version is more comprehensive and more useful!)

7 Likes

Thank you! I realized as I was falling asleep last night that I was neglectful and should have at least referenced and probably linked to your related helpful pinned post.

I think it can be helpful to present similar information in different ways, and I wasn't trying to usurp the place of your posts or anything. :slight_smile:

I do like your proposed documentation page, and I could see how it could be helpful to add something to this post about when using a private CA instead is a better idea. But I also was focusing on geographic block issues (since that's what's been coming up lately), and it's probably too long for people to read already. Not sure when I'll get the chance but I'll see if there's more to add here from your text at some point. (Or feel free to add it yourself, I made this a wiki for a reason.) Thanks.

7 Likes

13 posts were split to a new topic: Multi-Perspective Validation, simultaneous requests, and Apache caching

Interesting to note, Google only requires >50% success rate instead of 100%. The message they use if <50% is "X validation attempt(s) succeeded, Y validation attempt(s) failed. Must have more successful validation attempts than failed." Seems like they're currently using 6 total so 4+ are required to succeed.

1 Like

Let's Encrypt also doesn't require 100% of the secondary to succeed. Currently they allow one failure, and while they were ramping up to 5 locations I think it was temporarily two failures allowed.

4 Likes

2 posts were split to a new topic: Any changes to multi-perspective validations lately?