Questions re: Beginning Issuance from R3

That was an unfortunate and ill-considered design then.

3 Likes

And while I just gave this advice, it might not have actually given people much time to prepare. There was a notification on the blog on Sept. 17, but the only notice on the API Announcements category said that it might happen as soon as that day. So it may be that if you want to learn of new intermediates with enough time to add them to your systems that you need to watch both the API Announcements and the blog, though I don't see any good way to get notified of new blog posts (though one might be able to easily integrate with the RSS feed of it somehow).

3 Likes

Shouldn't be a matter anyway @petercooperjr, the integration guide is very clear:

For intermediate changes in particular, you should not hardcode the intermediate to use, but should use the Link: rel="up" header from the ACME protocol, since intermediates are likely to change.

3 Likes

when I first wrote a client several years ago, very very very few clients implemented this. i don't think certbot did this for quite some time either.

3 Likes

I would hope a client written many many years ago has seen some updates in the mean while?

You can't just pretend you're an ostrich and stick your head in the sand..

3 Likes

We are keeping an eye on things to monitor the extent of any breakage due to this change. So far, we have observed very few domains continuing to serve the X3 intermediate in their chain despite having gotten an R3-issued end-entity cert during their scheduled renewal. For those few domains where we have observed this failure mode (which is not necessarily a full breakage: browsers may still be able to verify the end-entity cert despite the wrong chain being served), we are working on determining commonalities so we can reach out to the appropriate ACME client developers.

Note, however, that even APIv1 ACME clients which do hard-code the /acme/issuer-cert API path rather than getting it from the <link rel="up"> attribute are not broken by this change: we changed the cert which is served at /acme/issuer-cert from X3 to R3. The only clients which would be broken by this change are those that bundle the intermediate cert directly, fetch it from a hardcoded non-ACME url (e.g. from the AIA url), or fetch the intermediate upon first issuance but then ignore it during renewals.

7 Likes

Indeed some DANE users had published TLSA records that match only X3. Since the announcement many have updated their TLSA records to also match R3/R4, but some have procrastinated. They'll have to make changes soon, it looks like X3 is now retired, and all new issuance is via R3.

A handful of domains monitored by the DANE survey in fact failed validation today and have been notified of the problem. Most are unaffected (use "3 1 1" records, or have already added R3/R4). Probably a few more procrastinators will only take action after a failure in the coming days...

3 Likes

3 posts were split to a new topic: Certify the Web Client Issues Possibly R3 Change

We're facing an issue with our IoT devices that were bundled with the intermediate certificate (X3) and when our server renew the certificate, they lost the ability to connect with server which prevent us from update them remotely.
Is it possible to request a certificate based on the intermediate X3 just to gave us a time window to update our devices? Or there's any procedure for recovery where we can request a certificate with that specific chain to Let's Encrypt?

2 Likes

What was your plan when X3 expired in 2021, or if they needed to bring the disaster recovery intermediate X4 online?

If your servers followed the best practice of renewing every 60 days, you should still have your older server certificate somewhere that was signed by X3 that should expire at some point in the next month. If you can restore that certificate and use it for your server, you should have a small window until it actually expires. I recommend you update your devices to chain to the ISRG root (and also include some other CAs rather than relying on Let's Encrypt being around forever) instead of a specific intermediate. Intermediates can change at any time.

You may also want to check out this thread by someone else who was looking at what to put in the trust store of their IoT devices, which has some additional ideas and cautions:

5 Likes

Thank you for your help and for the reference to that thread.
Regarding your first question, we updated the application so it can load multiple certificates into a cert pool, which is then used by the client it tries to establish a connection with server.
Regarding your suggestion to recover the older certificate, it wasn't possible, because the certificates are managed by cert-manager (on a kubernetes cluster) which replaces the older ones when their are renewed.

3 Likes

Luckily you've got a fairly recent backup from which you can pull the older certificate :wink:

3 Likes

Hi,
My Salesforce client makes a callout to one of our internal services which has the new R3 intermediate cert. The integration is broken now. Salesforce maintains the list of trusted root certificates. Since the intermediate cert has changed, should I contact Salesforce to get their certificate chain updated?

2 Likes

Welcome to the Let's Encrypt Community, Rittu :slightly_smiling_face:

That sounds wise to me, so long as you confirm the exact problem first. See the posts directly below from @Osiris and @rg305. Given that intermediate certificates can functionally change at any point in time, they need to keep things updated. They can use the following to get their information straight:

3 Likes

A TLS client shouldn't rely on intermediates in their trusted certificate store, but on root certificates. And the root certificate hasn't changed (yet). Therefore, it might be possible that your server isn't correctly configured or Salesforce is doing stupid things.

4 Likes

Short answer: No (not for that reason).
Longer answer: You might have to contact them if they are to blame for this problem (for other reasons).
Much Longer answer: You need to figure out where the problem is, so that you can get the right person to fix it.

3 Likes

I would first double-check that your service is sending the correct intermediate, before blaming Salesforce. (Though it's certainly possible that they are doing something wrong, I'm just saying to be doubly-sure that everything you're doing is right before heading in that direction.)

4 Likes

@ritturoy as @petercooperjr suggests, your internal service is not serving the full certificate chain. I'd go as far as to say this is 99% likely to be the problem, so fix it on the internal service. You could technically add the R3 intermediate into the trust store of whatever app is using the service so that your service doesn't need to serve the full chain, but that just moves the problem around.

4 Likes

Thank you folks! Like you all mentioned, the problem was with our internal service. It did not serve the new intermediate cert. We have a job that automatically renews the certs every three months. Somehow it did not get the new intermediate. We had to manually download it.

4 Likes

You probably have a client that hardcoded the intermediate certificate. You should ensure your client is able to adapt to changing intermediates, as this will likely break again in the future.

5 Likes