Impact
On June 11 2016 (UTC), we started sending an email to all active subscribers who provided an email address, informing them of an update to our subscriber agreement. This was done via an automated system which contained a bug that mistakenly prepended between 0 and 7,617 other email addresses to the body of the email. The result was that recipients could see the email addresses of other recipients. The problem was noticed and the system was stopped after 7,618 out of approximately 383,000 emails (1.9%) were sent. Each email mistakenly contained the email addresses from the emails sent prior to it, so earlier emails contained fewer addresses than later ones.
We apologize for the error. The following information is from the postmortem we performed in order to determine exactly how this happened and how we can prevent something like this from happening again.
Incident Timeline
- 11 Jun 2016 01:47 UTC - Started sending subscriber agreement update emails
- 11 Jun 2016 02:20 UTC - Received report that email contents may be problematic
- 11 Jun 2016 02:29 UTC - Email job terminated
- 11 Jun 2016 03:28 UTC - Preliminary public disclosure posted
How this Happened
A small piece of software had been written to handle one-off mass emailing to our subscribers. It was being used for the first time when this incident occurred.
The software went through code review and testing as it was being developed, but testing was insufficient. It did not catch a bug which prepended the email addresses of prior recipients to the body of emails.
Insufficient testing is considered to be the root cause of this incident.
What We Learned
Let’s Encrypt has high standards for review and testing of core CA systems, those related to the issuance and management of certificates. The mass emailing software was not considered to be part of our core CA systems because it does not deal with the issuance or management of certificates. As a result, it was developed outside of the controls and standards we have in place for core CA systems. Since the mass emailing software necessarily accesses subscribers’ confidential email addresses it should have been held to our established standards for development and testing.
Additionally, the sending of the emails should have been done in stages, with a very small number sent and checked for errors before ramping up. This is a strategy we employ for other systems and it would have further limited the bug’s impact.
Remediation
Our mass emailing software will be moving to boulder (our core CA software) where it can re-use well-tested email send functionality used for certificate expiration reminders. Core CA software development and testing guidelines will apply. More tests will be written, dry run and incremental send capabilities will be added and/or improved.
In the future, any software which has access to subscriber email addresses will be considered core CA software, even if it doesn’t directly have to do with the issuance or management of certificates.
What Went Well
The emailing job was stopped after only 1.9% of emails had been sent.
Additional technical and communications staff were brought online quickly to help with the response.
A preliminary public disclosure was made within 90 minutes of the incident and new information was added as it was confirmed. Staff worked to spread awareness of this information through various methods.