Summary
On 2021-10-01 a new Baseline Requirements revision (Ballot SC48v2) went into effect stating that “the Fully-Qualified Domain Name or the FQDN portion of the Wildcard Domain Name MUST consist solely of Domain Labels that are P-Labels or Non-Reserved LDH Labels”.
Let’s Encrypt had reviewed the requirement before the effective date, but missed a case to forbid a Reserved LDH Label when a hyphen is its second character. The code incorrectly allowed domains like a---foo.example.com but correctly forbade names like ab--foo.example.com.
On 2021-10-11 it was reported that Let’s Encrypt had issued some certificates that were not compliant with this new requirement. Let’s Encrypt verified the claim, and stopped CA issuance while a fix was deployed. An audit of certificates issued since 2021-10-01 revealed 7 affected certificates. The certificates were revoked within 24 hours of the report.
Timeline
Before Incident:
- 2021-07-15 15:00 UTC: SC48v2 voting period begins
- 2021-07-21 16:15 UTC: A Let’s Encrypt engineer files an internal ticket to review the Boulder CA software relevant to the proposed changes in SC48v2 and begins reviewing the ballot language and source code.
- 2021-07-21 21:05 UTC: The review is completed and the investigator concludes that the ballot language and source code match. Two additional engineers review the conclusion and agree.
- 2021-07-22 15:00 UTC: SC48v2 passes and the effective date will be 2021-10-01
- 2021-07-28 15:12 UTC: A Let’s Encrypt engineer closes the internal ticket with the conclusion that the Boulder CA software does not need updates
Incident:
- 2021-10-11 21:11 UTC: The report is sent to cert-prob-reports.
- 2021-10-11 21:47 UTC: A Let’s Encrypt engineer reviews the report and the team begins to evaluate it.
- 2021-10-11 22:10 UTC: Let’s Encrypt halts issuance to prevent possible, additional mis-issuance while the report is verified and confirmed.
- 2021-10-11 22:40 UTC: Let’s Encrypt confirms that the Boulder CA software does not comply with the Baseline Requirements once SC48v2 is in effect. The team begins to write a fix and review issued certificates for non-compliance.
- 2021-10-11 23:31 UTC: The fix is merged.
- 2021-10-12 00:06 UTC: The fix is deployed to the Let’s Encrypt staging environment and tested.
- 2021-10-12 00:54 UTC: The fix is deployed to the Let’s Encrypt production environment and issuance is restored.
- 2021-10-12 19:15 UTC: An audit of certificates issued since 2021-10-01 reveals 7 affected certificates. Affected subscribers were notified and the 7 certificates were revoked less than 24 hours after the problem report.
We have posted the full details to Mozilla Bugzilla: 1735247 - Let's Encrypt: Mis-issued certificates related to SC48v2