ACME v1/v2: Validating challenges from multiple network vantage points

On Wednesday February 19th, 2020 we’ll turn on stricter validation requirements in production. We’ll make multiple validation requests from different network perspectives. Most issuance should continue as normal; we believe that a small number of domain names may need fixing. The most common issue will be hosts that use extremely strict firewall rules to allow validation from only specified IP addresses (a practice we do not recommend).

Previously only one validation request from one of our primary datacentres was required. After Feb 19th we will make four total validation requests (1 from a primary datacentre, and 3 from remote datacentres). The primary request and at least 2 of the 3 remote requests must receive the correct challenge response value for the domain to be considered authorized.

In the future we will continue to evaluate adding more network perspectives and may change the number and required threshold.

Testing

This is the production deployment of the change we previously announced for our staging environment.

Testing that domain validation succeeds in the staging environment is the best way to determine if your ACME integration will be affected by this change. If you are currently successfully performing issuance in the staging environment there is no need for action on your part.

Please note this change affects both the ACME V2 API, and the deprecated ACME v1 API.

Exception List

We recognize that some ACME client deployments have problems satisfying multiple challenge requests e.g. due to unsynchronized DNS zones, inappropriate firewall rules, or because challenge responses are deprovisioned after the first request counter to RFC 8555.

To aid in a gradual roll-out of the new multiple validation requirement we will be deploying a temporary exception list. ACME Account IDs and domain names on this list will only require the primary datacentre request to succeed, maintaining our pre-existing validation strategy for those entries.

We will initially populate this list with domains that we anticipate, based on our logs, will have trouble with multi-perspective validation. We will only do this in the case where the associated ACME account has specified contact information to allow communicating that they’re on the temporary exception list.

If you have tested your integration in the staging environment and have found incompatibility with multi-perspective validation you may request to be added to the temporary exception list with this Google form: https://forms.gle/9QN7dxALJVAoRjMKA

This is strictly a temporary measure and on June 1st, 2020 we will be removing the exception list entirely.

16 Likes
During secondary validation: Incorrect TXT record
Testing renewal
Multi-VA implementation e-mails
Cert order fails with error "During secondary validation: Incorrect TXT record"
What is the current status of the implementation of multi-viewpoint validation?
Action required for v2 validation email
Authorization issues
Action required: New feature and your Let's Encrypt integration
Validation: DNS problem
Unable to renew time out
Certificate renew stopped working
Exception Request
Problem with verification
Certificate renewal failing with HTTP 200
Error when I try to generate certificate with traefikv2 acme tls challenge - Docker Swarm
Unable to renew cert
Problem during secondary verification
Timeout during connect (likely firewall problem)
Enable SSL for subdomain on another host
Unable to renew cert, to create cert
Servfail only on staging
During secondary validation: DNS problem: query timed out looking up A
CURL the challenge OK but `Invalid response` with certbot
Problem requst new and renew letsencrypt
Status of multi-va exception list
Failed http-01 challenge
Yet another "Timeout" while verifying via HTTP
The way to connect Lets Encrypt when some of it's IPs are blocked by a FireWall
During secondary validation: No valid IP addresses found for sjvapzr2.app-staging.havoc.sh
Timeouts using certbot despite 200 OK responses
Tls-alpn-01 with acme4j -> unauthorized
Timeout during connect (likely firewall problem)
"During secondary validation" error
[SOLVED] Unable to pass HTTP-01 verification "Error getting validation data"
During secondary validation: No valid IP addresses found
Certbot-auto renew failing to work after upgrading from 1.0.0 to 1.2.0
Remote IPs for specific renewal
Where Let's Encrypt Leads, Others Follow
HTTP challenges retry
Challenge failed but apache logs indicate success for the validation 200
Difficulty issuing for shintajim.ir
Can't renew During secondary validation: Fetching xxx Connection reset by peer
Unable to renew certificates
Whitelist hostnames for certbot validation?
Validation from ip with security issues
Failed Authorization (Timeout) But LetsEncrypt was able to access the server?!?
HTTD - Certbot fails to obtain a certificate
Webroot challenges now coming from amazonaws.com
Timeout during connect (likely firewall problem)

I've updated our original API announcement to include details on how to request being added to the temporary exception list:

4 Likes

This change is now live in the production environment. For more information about multi-perspective validation and how we use it to protect the security and integrity of Web PKI, check out our latest blog post!

12 Likes

We have removed the temporary exception list.

8 Likes

Correction: We granted three large integrators' request of an extension to June 22, so there are currently 3 accounts on the exception list. We plan to remove the list entirely on June 22.

6 Likes

The remaining exceptions have been removed and we are enforcing multi-perspective validation for all clients.

7 Likes