At 12:45 UTC we received a report to our cert-prob-reports@letsencrypt.org contact address that Let’s Encrypt was improperly handling CAA records with mixed case tags, resulting in mis-issuance under the baseline requirements. Thanks to Corey Bonnell of TrustWave for the report.
RFC 6844 Section 5.1 says: “Matching of tag values is case insensitive.” Let’s Encrypt’s implementation of CAA record processing processed CAA tags case sensitively. This led to CAA tags that were not lowercase being ignored during CAA validation.
The problem was quickly confirmed, and a fix was developed and reviewed, with tests, by 13:28 UTC - under an hour from the initial report. We deployed the fix to our staging environment at 14:37 UTC. We disabled issuance of new certificates in our production environment at 14:45 UTC, to prevent additional misissuance from occurring. We deployed the fix to our production environment at 15:20 UTC.
Our logging of the CAA records processed does not provide the case information we need to determine whether other issuances were affected by this bug. We plan to perform these two post-incident remediation items to start with:
- Improving the CAA validation logging in Boulder to log CAA records prior to our processing.
- Performing a scan of current CAA records for the domain names we have issued for in the past 90 days, specifically looking for tags in CAA records with non-lowercase characters. We’ll examine such instances on a case-by-case basis to determine the appropriate action.
The original reporter identified one certificate (https://crt.sh/?id=469407542) that was issued in violation of the CAA RFC as part of their testing. We have revoked this certificate as of 15:41 UTC.