We've noticed a surge in certificate requests for very long domain names (e.g., 10 DNS labels) that we believe are the result of unintended feedback loops between Caddy or autocert and Internet scanning tools. We'll be blocking some certificate requests, and may additionally pause accounts that show a large volume of this kind of issuance. Most Let's Encrypt subscribers will be unaffected, but read more if you use golang.org/x/crypto/acme/autocert or Caddy with On-Demand TLS.
The affected web servers automatically request certificates when a client initiates a TLS handshake containing a hostname for which the server doesn't have a certificate. This is problematic because any third party can cause the web server to request a certificate, sometimes without even intending to do so. And often for a hostname that will never be used again. This in turn wastes Let's Encrypt resources that could otherwise be spent issuing certificates for hostnames that will see active use.
For instance, this behavior can easily be triggered by researchers scanning the Internet. One behavior we've seen in particular: some affected servers will request hostnames of the form update.update.update.update.update.update.update.update.example.com, or git.git.git.git.git.example.com. We suspect this is triggered by researchers scanning CT logs, extracting hostnames, and making requests to those hostnames with "interesting" subdomains like update or git prepended. Those requests trigger issuance, which puts new hostnames in CT logs, which get the same treatment, resulting in hostnames like the above.
Caddy has some built-in defense against this behavior: users must enable restrictions for On-Demand TLS. For instance, one type of restriction is configuring an ask endpoint that answers the question of whether Caddy should request issuance. However, based on our observed client behavior, we suspect there are a number of hosts with an ask endpoint configured to simply return 200 for all hostnames.
Our recommendation to Caddy users is to disable On-Demand TLS, or to make sure the ask endpoint only answers 200 for a specific set of pre-approved subdomains (and not deeply nested subdomains).
Our recommendation to autocert users is to use a HostPolicy that allows issuance for a specific set of pre-approved subdomains (and not deeply nested subdomains). For instance, the policy provided by HostWhitelist will achieve this.
If you need certificates that work for many subdomains at once, wildcard certificates are a great option: they'll cover all subdomains one level deep from a base domain. Using a wildcard certificate will provide more reliability and faster service for your website, since your visitors won't need to wait on new certificate issuance each time they visit a new subdomain.
Over the next few weeks we'll be deploying heuristics to block requests that seem to indicate unrestricted On-Demand TLS; for instance, requests that contain identical sequential domain labels from a specific list. If you run into requests that are blocked but shouldn't be, please post in the Let's Encrypt Community Forum.