Don't forget the part about OCSP URIs in the intermediates, as they are gone for R3/R4 and E1/E2 as far as I remember.
5 Likes
Hello. Sorry for my noob question.
Is there any chance for the new certificate (R3) to support the old (X3) intermediate cert? Maybe somehow via alternate URL or something like that? We have a problem with some of our clients who hardcoded the certificates issuer footprint on their end, so they now can't establish a connection with the new certificates.
2 Likes
No, that's not really how it works. Any system getting a new certificate needs to also download the intermediate that it was signed with and configure the server to serve it alongside the certificate. Anyone who is trying to validate that a certificate is actually signed by Let's Encrypt needs to check the full certificate chain and validate it against a root certificate, not any specific intermediate certificate. While they did just change the intermediate for the first time in a long while, it could have changed at any point and can change again at any point (such as if the disaster-recovery intermediates start getting used).
If you really need to track the intermediates for some reason (which I guess maybe some DANE implementations do?) then you need to subscribe to the API Announcements to get notified about upcoming new intermediates so you can be proactive about adding them to your system. But there are several that could be used so you'd need to be prepared for certificates that are signed by any of them.
7 Likes
Thanks for the clarification. 
2 Likes
That was an unfortunate and ill-considered design then.
4 Likes
And while I just gave this advice, it might not have actually given people much time to prepare. There was a notification on the blog on Sept. 17, but the only notice on the API Announcements category said that it might happen as soon as that day. So it may be that if you want to learn of new intermediates with enough time to add them to your systems that you need to watch both the API Announcements and the blog, though I don't see any good way to get notified of new blog posts (though one might be able to easily integrate with the RSS feed of it somehow).
4 Likes
Shouldn't be a matter anyway @petercooperjr, the integration guide is very clear:
For intermediate changes in particular, you should not hardcode the intermediate to use, but should use the
Link: rel="up"header from the ACME protocol, since intermediates are likely to change.
4 Likes
when I first wrote a client several years ago, very very very few clients implemented this. i don't think certbot did this for quite some time either.
4 Likes
I would hope a client written many many years ago has seen some updates in the mean while?
You can't just pretend you're an ostrich and stick your head in the sand..
4 Likes
We are keeping an eye on things to monitor the extent of any breakage due to this change. So far, we have observed very few domains continuing to serve the X3 intermediate in their chain despite having gotten an R3-issued end-entity cert during their scheduled renewal. For those few domains where we have observed this failure mode (which is not necessarily a full breakage: browsers may still be able to verify the end-entity cert despite the wrong chain being served), we are working on determining commonalities so we can reach out to the appropriate ACME client developers.
Note, however, that even APIv1 ACME clients which do hard-code the /acme/issuer-cert API path rather than getting it from the <link rel="up"> attribute are not broken by this change: we changed the cert which is served at /acme/issuer-cert from X3 to R3. The only clients which would be broken by this change are those that bundle the intermediate cert directly, fetch it from a hardcoded non-ACME url (e.g. from the AIA url), or fetch the intermediate upon first issuance but then ignore it during renewals.
8 Likes
Indeed some DANE users had published TLSA records that match only X3. Since the announcement many have updated their TLSA records to also match R3/R4, but some have procrastinated. They'll have to make changes soon, it looks like X3 is now retired, and all new issuance is via R3.
A handful of domains monitored by the DANE survey in fact failed validation today and have been notified of the problem. Most are unaffected (use "3 1 1" records, or have already added R3/R4). Probably a few more procrastinators will only take action after a failure in the coming days...
3 Likes
We're facing an issue with our IoT devices that were bundled with the intermediate certificate (X3) and when our server renew the certificate, they lost the ability to connect with server which prevent us from update them remotely.
Is it possible to request a certificate based on the intermediate X3 just to gave us a time window to update our devices? Or there's any procedure for recovery where we can request a certificate with that specific chain to Let's Encrypt?
2 Likes
What was your plan when X3 expired in 2021, or if they needed to bring the disaster recovery intermediate X4 online?
If your servers followed the best practice of renewing every 60 days, you should still have your older server certificate somewhere that was signed by X3 that should expire at some point in the next month. If you can restore that certificate and use it for your server, you should have a small window until it actually expires. I recommend you update your devices to chain to the ISRG root (and also include some other CAs rather than relying on Let's Encrypt being around forever) instead of a specific intermediate. Intermediates can change at any time.
You may also want to check out this thread by someone else who was looking at what to put in the trust store of their IoT devices, which has some additional ideas and cautions:
6 Likes
Thank you for your help and for the reference to that thread.
Regarding your first question, we updated the application so it can load multiple certificates into a cert pool, which is then used by the client it tries to establish a connection with server.
Regarding your suggestion to recover the older certificate, it wasn't possible, because the certificates are managed by cert-manager (on a kubernetes cluster) which replaces the older ones when their are renewed.
3 Likes
Luckily you've got a fairly recent backup from which you can pull the older certificate 
4 Likes
Hi,
My Salesforce client makes a callout to one of our internal services which has the new R3 intermediate cert. The integration is broken now. Salesforce maintains the list of trusted root certificates. Since the intermediate cert has changed, should I contact Salesforce to get their certificate chain updated?
2 Likes
Welcome to the Let's Encrypt Community, Rittu 
That sounds wise to me, so long as you confirm the exact problem first. See the posts directly below from @Osiris and @rg305. Given that intermediate certificates can functionally change at any point in time, they need to keep things updated. They can use the following to get their information straight:
4 Likes
A TLS client shouldn't rely on intermediates in their trusted certificate store, but on root certificates. And the root certificate hasn't changed (yet). Therefore, it might be possible that your server isn't correctly configured or Salesforce is doing stupid things.
5 Likes
Short answer: No (not for that reason).
Longer answer: You might have to contact them if they are to blame for this problem (for other reasons).
Much Longer answer: You need to figure out where the problem is, so that you can get the right person to fix it.
4 Likes