Feedback wanted: DST Root CA X3 expiration all-subscriber email

[jsha]

Hi all,

DST Root CA X3 is expiring in September. As described in Providing a longer certificate chain by default, we plan to start serving the Android-compatible "long chain" well before then, so most people don't need to worry about compatibility issues. Still, it's an important enough transition point that we want to send an all-subscriber email letting people know it's coming up. Here's our draft email so far. We've always gotten great feedback from forum members on improving our emails, so I'd like to know what you all think of this one.

Thanks,
Jacob

---

Hello from the staff at Let’s Encrypt. We're sending you this email because you use Let's Encrypt certificates. For instance, our records show you have a certificate for {{hostname}}. If you're receiving this in error, please see the unsubscribe link below.

On September 30, there will be a small change in how older browsers and devices trust Let's Encrypt certificates. If you run a typical website, you won't notice a difference - the vast majority of your visitors will still accept your Let's Encrypt certificate. If you provide an API or have to support IoT devices, you might have to pay a little more attention to the change.

Let's Encrypt has a "root certificate" called ISRG Root X1. Modern browsers and devices trust the Let's Encrypt certificate installed on your website because they include ISRG Root X1 in their list of root certificates. To make sure the certificates we issue are trusted on older devices, we also have a "cross-signature" from an older root certificate: DST Root CA X3.

When we got started, that older root certificate (DST Root CA X3) helped us get off the ground and be trusted by almost every device immediately. The newer root certificate (ISRG Root X1) is now widely trusted too - but some older devices won't ever trust it because they don't get software updates (for example, an iPhone 4 or an HTC Dream). See https://letsencrypt.org/docs/certificate-compatibility/ for a list of which platforms trust ISRG Root X1.

DST Root CA X3 will expire on September 30, 2021. That means those older devices that don't trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let's Encrypt certificates. There's one important exception: older Android devices that don’t trust ISRG Root X1 will continue to work with Let's Encrypt, thanks to a special cross-sign from DST Root CA X3 that extends past that root's expiration. This exception only works for Android.

What should you do? For most people, nothing at all! We've set up our certificate issuance so your web site will do the right thing in most cases, favoring broad compatibility. If you provide an API or have to support IoT devices, you'll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later1. In OpenSSL 1.0.x, a quirk in certificate verification means that even clients that trust ISRG Root X1 will fail when presented with the Android-compatible certificate chain we are recommending by default.

If you have any questions, we recommend asking on our community support forums: https://community.letsencrypt.org/ because this is a no-reply account and you will not get a response via email.

Since 2015 we’ve served the world with 1.6 billion free certificates, each one providing security and privacy to people on the Web. It’s work that’s 100% funded by charitable donations since we are a nonprofit. If your company is interested in sponsorship, please email sponsor@letsencrypt.org. If you can make a donation, we ask that you consider supporting our work today. https://letsencrypt.org/donate/. Thank you.

• The Let’s Encrypt team

If you are receiving this email in error, unsubscribe at

[schoen]

Maybe add a "How to check what your client did"/"How to check what chain your site is currently using"?

It seems like apart from the challenge of people understanding this properly, there will be a challenge about people who aren't sure whether their clients did or didn't do the right thing (and some of them may be right to wonder!).

[schoen]

(I'm wondering whether it's worth being a little more explicit that there may be malfunctioning clients or control panels that hard-code the chain even without being told to prefer a particular chain... this is potentially helpful information to the minority of people using such a client, but may be confusing or distracting to the majority who don't.)

[griffin]

As I see the idea behind this email as a heads-up of potential issues, perhaps it might be prudent to put a separate, dedicated paragraph or box that explicitly explains what to look out for.

[petercooperjr]

Generally looks pretty good, but of course I can find things to nitpick about.

@jsha

certificate for {{hostname}}

Is this just going to show one hostname per account? If I have several hostnames, and only get an email relating to one, it may be confusing as I might think I only need to pay attention to that one host. I'm not sure of the best way to work around this, as I'm sure that listing all the hostnames for an account would also be terrible for some users, but maybe be clearer that the account may have other hostnames and it's something affecting all Let's Encrypt certificates?

@jsha

If you're receiving this in error

I think more likely the message needs to be that if they're not understanding it they should forward it to whatever technical person is maintaining their web site. (I mean I see why you need to reference unsubscribing as well, but I don't know how often someone is listed as the contact but has no idea what a Let's Encrypt is.)

@jsha

If you provide an API or have to support IoT devices

I don't know how technical an audience this is going for, but using these acronyms without defining them might confuse some people (though more likely to confuse the people who don't need to worry about it). I guess I'm just thinking that there's a lot of jargon associated with certificates already, so avoiding any more jargon than one needs to might be helpful. Here you're using terms that aren't even in the official glossary.

@jsha

If you provide an API or have to support IoT devices, you'll need to make sure of two things:

For the second step here of using OpenSSL, the other option is to have their ACME client use the alternate chain, right? I'm thinking that in many of these scenarios it'd be easier to get the server's ACME client to use the alternate chain rather than getting the embedded-device-type client to upgrade their version of OpenSSL. I'm not sure of the best way of wording it, but I think that "use the alternate chain if your clients don't handle a chain with expired roots" needs to be in there somewhere, even if it's just a footnote or a page in the documentation that this links to or whatever.

@jsha

If you have any questions, we recommend asking on our community support forums

I don't know if this email should be linking to the forums in general, rather than to a specific forum thread or a help page of some sort. If it turns out that there's some other thing one wishes got put in the email (or there's some other update to the transition or whatever) it may be nice to there to be a good "landing page" with any updates.

@schoen

(I'm wondering whether it's worth being a little more explicit that there may be malfunctioning clients or control panels that hard-code the chain even without being told to prefer a particular chain... this is potentially helpful information to the minority of people using such a client, but may be confusing or distracting to the majority who don't.)

Yeah, this message is very focused on the Sept. 30 date, but when I first saw the thread I originally assumed that this would be informing people of the May 4 date when any client that's not expecting multiple certs in the chain will fail. Though obviously both are important for people to know about.

Some other thoughts:

• Is this getting localized into various languages?
• I definitely agree with the above comments that "How do I test this?" and "How do I know what to look for" should probably get addressed. Maybe there should be some mention of the staging environment, though the message is already getting really long as it is.

[webprofusion]

It may be useful to put a general version number to the 'older Android' line as that could instead be ambiguously interpreted as devices from last year (for instance) and cause a panic

[webprofusion]

Also, knowing roughly when this email will go out would be useful for me as I'll get a spike in support requests to explain what it all means to my ACME client users.

[bruncsak]

It is very important information to transmit to the users. However, because of its length it may distract many of them from reading the e-mail. It would be preferable to send shorter, but well focused message (executive summary kind), and have a link in the e-mail, which explains all the technical details behind for the interested parties. This page may be updated or expanded any time, if new information or need for correction arises.

[tialaramex]

Not directly related to this email, but to the ancillary information referenced. It would be good to establish exactly what will happen on other popular platforms, by setting clocks ahead on a test system. Unfortunately because of the very reasonable policy of Let's Encrypt to rely on their full automation issuance, we won't be able to conduct good tests until at least 90 days before the actual problems would arise, because that's the first point where we can show a test system a certificate which is valid after the old DST Root expires.

The desired test scenario is: Test system believes [due to clock setting] DST Root CA X3 expired (say) yesterday, and that the leaf certificate it is being shown won't expire until (say) tomorrow. Try it against servers which during TLS handshake present (as their "chain"):

1. Just the leaf certificate
2. Default "full chain" Let's Encrypt plan to offer
3. Alternate "full chain" planned
4. Any spontaneous suggestions or proposed alternate chains people think might be a good idea

For modern browsers none of these should make any difference, the browsers are smart enough to decide these certificates are in fact trustworthy. I guess an airgapped system in some cases might require a chain that gets it to ISRG Root X1 (but in Firefox even that isn't necessary). But for a lot of the older systems listed like older Java or video game consoles, and doubtless many we've never documented, these are likely to cause different behaviour.

Having a few months to nail down what can work, what can't work for different platforms means here on this community site we can help people who struggle after the real deadline, and we'll know if there's no hope for their scenario and can send them to another CA without wasting a lot of their time. We might even find there's a situation where a different chain is valuable, in time to have Let's Encrypt offer that to real users or instructions to be written for how to DIY assemble your own chain file in these corner cases.

I'm writing about this here because I think this sort of thing needs some preparation, we cannot run these tests now (because of the Let's Encrypt policies) but we can plan to run them in early July I think.

[petercooperjr]

Well, the staging environment is set up to chain to an expired root, to emulate the future expiration of DST Root X3 as best as it can. So if one can set up a complete test environment, including having the appropriate fake version of the certificates in your test trust store, then there's certainly some testing that can happen. But of course nothing is exactly like production, and probably a lot of those "older systems" don't seem to really have easily-updated trust stores (which is why we're in this mess in the first place).

[jsha]

@petercooperjr

Is this just going to show one hostname per account?

Perhaps... We've also in the past shown a small handful of hostnames, with "and N more...".

@petercooperjr

I think more likely the message needs to be that if they're not understanding it they should forward it to whatever technical person is maintaining their web site.

Ah, good point!

@petercooperjr

I guess I'm just thinking that there's a lot of jargon associated with certificates already, so avoiding any more jargon than one needs to might be helpful. Here you're using terms that aren't even in the official glossary.

@petercooperjr

For the second step here of using OpenSSL, the other option is to have their ACME client use the alternate chain, right?

Yep. I'll think about a way to make this clearer.

@petercooperjr

Is this getting localized into various languages?

The email: No. We don't have a mapping of email addresses to languages, and we also don't have the tooling to do multiple variants of an email. However, if we follow @bruncsak's excellent advice lower in the thread and put most of the information on a web page, we have the option to provide translations (depending on availability of our wonderful volunteer translators).

@webprofusion

It may be useful to put a general version number to the 'older Android'

@webprofusion

Also, knowing roughly when this email will go out would be useful for me as I'll get a spike in support requests to explain what it all means to my ACME client users.

Right now my estimate is "sometime in the next month, probably around mid-April" but I don't want to commit to a firm date yet since it depends on availability of my colleagues. We'll plan on sending out a small trickle at first, seeing what sort of input we get from users, and then sending out the rest.

@tialaramex

It would be good to establish exactly what will happen on other popular platforms, by setting clocks ahead on a test system. Unfortunately because of the very reasonable policy of Let's Encrypt to rely on their full automation issuance, we won't be able to conduct good tests until at least 90 days before the actual problems would arise, because that's the first point where we can show a test system a certificate which is valid after the old DST Root expires.

Yes, very much agreed. We will help out with such tests when we get to July, and any help the community can provide by testing on a variety of older devices and so on will be quite useful!

Thanks,
Jacob

[tlrenkensebastian]

I completely agree with this advice. And in keeping with the other suggestions above, maybe having a post be stickied at the top of the Help forum titled, "Additional info on the DST Root CA X3 expiration."

[yschimke]

I'd suggest linking to pre-seeded forum posts instead of just the community forums. Will help avoid a lot of redundant questions.

[jsha]

Good point. I've started a thread here: Help thread for DST Root CA X3 expiration (September 2021), and will link to it from the web page (which will be linked from the email).