Questions re: Beginning Issuance from R3

Yep... R3 is a go!

2 Likes

Remember that all conversations about the expiration of DST Root CA X3 (and therefore the expiration of our X3 and R3 cross-signed intermediates) are really conversations about the upcoming Chain Switch, not conversations about the switch from X3 to R3. If we had not done this changeover today, all the same issues regarding expiration in mid-late 2021 would still exist with X3, so it's an orthogonal issue. We do plan to change our default chain on Jan 11th, and disallow issuance from the cross-signed R3 entirely in early summer. But please keep conversations about those events (and their necessity) to other threads, so that this one can remain clear and concise for folks who have questions about today's X3-to-R3 switch.

5 Likes

The specifics of the confusion (from which this very topic came) can be found here:

2 Likes

Ah, Discourse didn't give any hint of new posts in that topic. I guess the API announcement section is regarded as a regular section and I should have manually selected notifications on new posts...

Also, I'm wondering how many clients are out there with a hardcoded intermediate now setting a broken chain. No new threads up till now about that..

3 Likes

We're wondering the same thing! That's one reason for the announcement thread: so that someone whose site seems to be broken by this change can see the announcement and our explanation of exactly this failure mode there.

But we're also spot-checking domains which have renewed their certs since this changeover to see if they've properly updated their full chain, and trying to identify commonalities (acme clients, hosting providers, etc) between any such domains.

7 Likes

So now that R3 is live in production (hooray!), I still don't see any changes in staging. (Or at least, a staging certificate that I just issued and one I tried a couple weeks ago are both signed by the same "Fake LE Intermediate X1".) Is there still a change expected in staging relating to this change?

This guess of mine was apparently wrong, as the CPS section 7.1 still says that the CN of intermediates will be Let's Encrypt Authority X<n>. Is this a problem, or is that part of the CPS not actually authoritative but just describing the initial intermediates and not necessarily all active intermediates?

4 Likes

We decided not to change the staging certificate yet. We will probably sign a new root and intermediates for staging tests when we start testing our multi-issuer support (rsa and ecdsa roots), but that is still a while away.

Good catch! I'll make sure we get that updated.

5 Likes

Don't forget the part about OCSP URIs in the intermediates, as they are gone for R3/R4 and E1/E2 as far as I remember.

5 Likes

Hello. Sorry for my noob question.
Is there any chance for the new certificate (R3) to support the old (X3) intermediate cert? Maybe somehow via alternate URL or something like that? We have a problem with some of our clients who hardcoded the certificates issuer footprint on their end, so they now can't establish a connection with the new certificates.

2 Likes

No, that's not really how it works. Any system getting a new certificate needs to also download the intermediate that it was signed with and configure the server to serve it alongside the certificate. Anyone who is trying to validate that a certificate is actually signed by Let's Encrypt needs to check the full certificate chain and validate it against a root certificate, not any specific intermediate certificate. While they did just change the intermediate for the first time in a long while, it could have changed at any point and can change again at any point (such as if the disaster-recovery intermediates start getting used).

If you really need to track the intermediates for some reason (which I guess maybe some DANE implementations do?) then you need to subscribe to the API Announcements to get notified about upcoming new intermediates so you can be proactive about adding them to your system. But there are several that could be used so you'd need to be prepared for certificates that are signed by any of them.

6 Likes

Thanks for the clarification. :cry:

2 Likes

That was an unfortunate and ill-considered design then.

3 Likes

And while I just gave this advice, it might not have actually given people much time to prepare. There was a notification on the blog on Sept. 17, but the only notice on the API Announcements category said that it might happen as soon as that day. So it may be that if you want to learn of new intermediates with enough time to add them to your systems that you need to watch both the API Announcements and the blog, though I don't see any good way to get notified of new blog posts (though one might be able to easily integrate with the RSS feed of it somehow).

3 Likes

Shouldn't be a matter anyway @petercooperjr, the integration guide is very clear:

For intermediate changes in particular, you should not hardcode the intermediate to use, but should use the Link: rel="up" header from the ACME protocol, since intermediates are likely to change.

3 Likes

when I first wrote a client several years ago, very very very few clients implemented this. i don't think certbot did this for quite some time either.

3 Likes

I would hope a client written many many years ago has seen some updates in the mean while?

You can't just pretend you're an ostrich and stick your head in the sand..

3 Likes

We are keeping an eye on things to monitor the extent of any breakage due to this change. So far, we have observed very few domains continuing to serve the X3 intermediate in their chain despite having gotten an R3-issued end-entity cert during their scheduled renewal. For those few domains where we have observed this failure mode (which is not necessarily a full breakage: browsers may still be able to verify the end-entity cert despite the wrong chain being served), we are working on determining commonalities so we can reach out to the appropriate ACME client developers.

Note, however, that even APIv1 ACME clients which do hard-code the /acme/issuer-cert API path rather than getting it from the <link rel="up"> attribute are not broken by this change: we changed the cert which is served at /acme/issuer-cert from X3 to R3. The only clients which would be broken by this change are those that bundle the intermediate cert directly, fetch it from a hardcoded non-ACME url (e.g. from the AIA url), or fetch the intermediate upon first issuance but then ignore it during renewals.

7 Likes

Indeed some DANE users had published TLSA records that match only X3. Since the announcement many have updated their TLSA records to also match R3/R4, but some have procrastinated. They'll have to make changes soon, it looks like X3 is now retired, and all new issuance is via R3.

A handful of domains monitored by the DANE survey in fact failed validation today and have been notified of the problem. Most are unaffected (use "3 1 1" records, or have already added R3/R4). Probably a few more procrastinators will only take action after a failure in the coming days...

3 Likes

3 posts were split to a new topic: Certify the Web Client Issues Possibly R3 Change

We're facing an issue with our IoT devices that were bundled with the intermediate certificate (X3) and when our server renew the certificate, they lost the ability to connect with server which prevent us from update them remotely.
Is it possible to request a certificate based on the intermediate X3 just to gave us a time window to update our devices? Or there's any procedure for recovery where we can request a certificate with that specific chain to Let's Encrypt?

2 Likes