ACME v1/v2: Validating challenges from multiple network vantage points

On Wednesday February 19th, 2020 we’ll turn on stricter validation requirements in production. We’ll make multiple validation requests from different network perspectives. Most issuance should continue as normal; we believe that a small number of domain names may need fixing. The most common issue will be hosts that use extremely strict firewall rules to allow validation from only specified IP addresses (a practice we do not recommend).

Previously only one validation request from one of our primary datacentres was required. After Feb 19th we will make four total validation requests (1 from a primary datacentre, and 3 from remote datacentres). The primary request and at least 2 of the 3 remote requests must receive the correct challenge response value for the domain to be considered authorized.

In the future we will continue to evaluate adding more network perspectives and may change the number and required threshold.

Testing

This is the production deployment of the change we previously announced for our staging environment.

Testing that domain validation succeeds in the staging environment is the best way to determine if your ACME integration will be affected by this change. If you are currently successfully performing issuance in the staging environment there is no need for action on your part.

Please note this change affects both the ACME V2 API, and the deprecated ACME v1 API.

Exception List

We recognize that some ACME client deployments have problems satisfying multiple challenge requests e.g. due to unsynchronized DNS zones, inappropriate firewall rules, or because challenge responses are deprovisioned after the first request counter to RFC 8555.

To aid in a gradual roll-out of the new multiple validation requirement we will be deploying a temporary exception list. ACME Account IDs and domain names on this list will only require the primary datacentre request to succeed, maintaining our pre-existing validation strategy for those entries.

We will initially populate this list with domains that we anticipate, based on our logs, will have trouble with multi-perspective validation. We will only do this in the case where the associated ACME account has specified contact information to allow communicating that they’re on the temporary exception list.

If you have tested your integration in the staging environment and have found incompatibility with multi-perspective validation you may request to be added to the temporary exception list with this Google form: https://forms.gle/9QN7dxALJVAoRjMKA

This is strictly a temporary measure and on June 1st, 2020 we will be removing the exception list entirely.

16 Likes
During secondary validation: Incorrect TXT record
Testing renewal
Multi-VA implementation e-mails
Cert order fails with error "During secondary validation: Incorrect TXT record"
What is the current status of the implementation of multi-viewpoint validation?
Action required for v2 validation email
Action required: New feature and your Let's Encrypt integration
Unable to renew cert
Error when I try to generate certificate with traefikv2 acme tls challenge - Docker Swarm
Certificate renewal failing with HTTP 200
Problem with verification
Exception Request
Certificate renew stopped working
Unable to renew time out
Authorization issues
Validation: DNS problem
Timeout during connect (likely firewall problem)
Webroot challenges now coming from amazonaws.com
Validation from ip with security issues
Whitelist hostnames for certbot validation?
Unable to renew certificates
Difficulty issuing for shintajim.ir
Challenge failed but apache logs indicate success for the validation 200
Where Let's Encrypt Leads, Others Follow
Remote IPs for specific renewal
Certbot-auto renew failing to work after upgrading from 1.0.0 to 1.2.0
Can't renew During secondary validation: Fetching xxx Connection reset by peer
Status of multi-va exception list
HTTD - Certbot fails to obtain a certificate
Failed Authorization (Timeout) But LetsEncrypt was able to access the server?!?
HTTP challenges retry
Problem requst new and renew letsencrypt
During secondary validation: No valid IP addresses found
During secondary validation: DNS problem: query timed out looking up A
Servfail only on staging
Unable to renew cert, to create cert
Enable SSL for subdomain on another host
Timeout during connect (likely firewall problem)
Problem during secondary verification

I’ve updated our original API announcement to include details on how to request being added to the temporary exception list:

4 Likes

This change is now live in the production environment. For more information about multi-perspective validation and how we use it to protect the security and integrity of Web PKI, check out our latest blog post!

12 Likes

We have removed the temporary exception list.

7 Likes

Correction: We granted three large integrators’ request of an extension to June 22, so there are currently 3 accounts on the exception list. We plan to remove the list entirely on June 22.

5 Likes

The remaining exceptions have been removed and we are enforcing multi-perspective validation for all clients.

6 Likes