Policy on Issuance Chain Changes

[dan-golioth]

Before the June 6th, 2024 change, ECDSA certificates could be issued from RSA intermediates. Now, as mentioned in the previously linked post, all ECDSA certificates are issued from the ECDSA intermediates, which by default will be signed by ISRG Root X1, with the option to opt-in to an all ECDSA chain where the intermediate is signed by ISRG Root X2.

From the first paragraph of the post:

We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate.

Notice of the date of the switchover to the new intermediates appears to have first been given on June 10th, 2023 (roughly a year in advance). From a chain of trust perspective, care was taken to ensure that users of ECDSA certificates would still have broad compatibility by making the default behavior to use ECDSA intermediates signed by X1. However, the notice that ECDSA certificates would only be issued from ECDSA intermediates does not appear to have been given until the April 12th, 2024 post, roughly 2 months before the switchover was set to occur.

My question is whether there was previous indication that ECDSA certificates would only be issued from ECDSA intermediates, rather than giving users the option to have them issued from RSA intermediates or ECDSA intermediates? If not, is 2 month notice considered to be acceptable by Let's Encrypt for this type of change moving forward?

While attention was called to compatibility issues for folks using the ill-advised strategy of pinning to intermediate certificates, there was no acknowledgement of how users relying on one root or the other could be impacted by the change in curves / algorithms used by intermediates. Not all clients support all curves / algorithms, so moving from, for example, an ECDSA -> RSA (intermediate) -> RSA (root) chain to an ECDSA -> ECDSA (intermediate) -> RSA (root) chain could result in breaking clients regardless of whether the root RSA certificate remained the same.

If this sort of change / notice period is aligned with Let's Encrypt policy, I would recommend noting so somewhere on letsencrypt.com (my apologies if I missed where this is already present, please feel free to link me to it). If it is not, I would recommend continuing to allow issuance of ECDSA certificates from RSA intermediates for a more lengthy period of time to allow for client migration.

Thank you!

[Nummer378]

I guess this depends a bit on how close you were following the development. The ECDSA chain was always intended to be the only issuer for ECDSA leafs. It was just hidden behind an allowlist for the past 3 years, but it was clear that this change would come eventually, ever since the ECDSA chain became a thing in 2021.

When the new certificates were announced in March 2024, it was hinted that the ECDSA allow list would (finally) go away soon (and hence result in every ECDSA leaf being signed with an ECDSA intermediate):

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

The final decision to abolish the allowlist was indeed only announced on April 12th. So I guess this was implicitly clear to inside folks, but perhaps not clearly communicated to the public.

@dan-golioth

Not all clients support all curves / algorithms, so moving from, for example, an ECDSA -> RSA (intermediate) -> RSA (root) chain to an ECDSA -> ECDSA (intermediate) -> RSA (root) chain could result in breaking clients regardless of whether the root RSA certificate remained the same.

Which client supports an ECDSA leaf certificate, but does not support an ECDSA intermediate? Could you name an example? Compatibility issues like this are - I believe - not expected.

@dan-golioth

If not, is 2 month notice considered to be acceptable by Let's Encrypt for this type of change moving forward?

Let's Encrypt has previously held the stance that intermediates can change at any time, for any reason. Let's Encrypt has always had backup certificates that could have been activated at any time, without any prior notice. Admittedly, this generally doesn't involve a change of cryptographic algorithms - but with future cryptographic agility in mind, this may become a more common reality.

[aarongable]

Hi @dan-golioth, and welcome to the forum!

You pose a good question, and I stand behind @Nummer378's answer. While I apologize that the communication of these changes did not live up to your expectations, the eventual issuance of all ECDSA end-entity certificates from our ECDSA intermediates was announced as early as 2021.

We do not make guarantees about when or how far in advance we will announce changes. The timelines depend on a multitude of factors, including but not limited to: new regulatory requirements, monetary costs, engineering roadblocks and project sequencing, our desire to do well by our Subscribers, and the kind of change being made.

In this case, as already mentioned, we expected the impact of doing away with the ECDSA Allow-List to be quite small. And based on the metrics and feedback collected in the two months since the change, so far that expectation has been met. Again, I apologize if this change has caused you any particular issues, and I hope our community is able to help you resolve those issues.

[dan-golioth]

@Nummer378 and @aarongable -- thank you for following up!

When the new certificates were announced in March 2024, it was hinted that the ECDSA allow list would (finally) go away soon (and hence result in every ECDSA leaf being signed with an ECDSA intermediate):

This may hint at the fact that the ECDSA allow-list could go away in the future, but I don't think that implies that there would be no option to use an RSA intermediate to sign an ECDSA leaf. I see a difference in enabling new functionality (i.e. being able to get get an ECDSA leaf from an ECDSA intermediate) and removing existing functionality (i.e. being able to get ECDSA leaf from an RSA intermediate), and I think it is important that both are communicated explicitly. I think the ideal scenario here would be to allow subscribers to choose between:

1. ECDSA intermediate with X1 root.
2. ECDSA intermediate with X2 root (all ECDSA chain -- can be selected by specifying preferred chain).
3. RSA intermediate with X1 root.

(1) and (2) were made possible, while (3) was not.

Which client supports an ECDSA leaf certificate, but does not support an ECDSA intermediate? Could you name an example? Compatibility issues like this are - I believe - not expected.

The most clear case of this would be a client that supported ECDSA P-256, but not ECDSA P-384. They could easily be broken by the move from an RSA intermediate with an ECDSA P-256 leaf to an ECDSA P-384 intermediate with an ECDSA P-256 leaf.

We do not make guarantees about when or how far in advance we will announce changes. The timelines depend on a multitude of factors, including but not limited to: new regulatory requirements, monetary costs, engineering roadblocks and project sequencing, our desire to do well by our Subscribers, and the kind of change being made.

Thanks for sharing this policy! I completely understand that there are constraints and unforeseen circumstances. Unfortunately that can lead to negative impacts on some subscribers, and it may be helpful to communicate this more explicitly in the form of a policy (even if the policy is quite lenient in the guarantees that are made).

Overall, Let's Encrypt provides a wonderful service that benefits countless folks on a daily basis. I believe there is possible room for improvement in this process, but I also have a great appreciation for the work that y'all do, and I understand that best efforts are being made. If there is anything I can do to help improve this process moving forward I would be happy to contribute.

Thanks again for your thoughtful feedback!

[petercooperjr]

In terms of practical workarounds for anyone having problems, I think that there is some general assumption that systems with incomplete or flaky ECDSA support would be using RSA leaf certificates too. And one of the great things about Let's Encrypt pushing for a standard protocol for CAs to use is that there are now several CAs offering free certificates using ACME, so if one needs an intermediate with requirements that Let's Encrypt can't provide then it may work to use one of those other CAs. (Though it may be helpful to describe the use case in more detail so that others can learn about what is and isn't compatible with what.)

[Nummer378]

@dan-golioth

This may hint at the fact that the ECDSA allow-list could go away in the future, but I don't think that implies that there would be no option to use an RSA intermediate to sign an ECDSA leaf.

But this is the behavior we had since 2021 - once your account was on the allowlist (i.e. you opt into the new feature) you were no longer able to get ECDSA leafs signed with RSA intermediates. So an allowlist removal implies exactly that: The opt-in becomes the default (and a opt-out never existed, the allowlist was always one-way only).

@dan-golioth

The most clear case of this would be a client that supported ECDSA P-256, but not ECDSA P-384. They could easily be broken by the move from an RSA intermediate with an ECDSA P-256 leaf to an ECDSA P-384 intermediate with an ECDSA P-256 leaf.

I agree, a client that does not support P-384 would be in trouble. I was curious if you know of a particular implementation that fulfills these criteria, and if so what type of client that is (e.g. embedded TLS client? IoT? Why doesn't it support P-384? Space-constrained?).

[griffin]

Myself, I take a lot of the feedback @dan-golioth has presented here as more about principles than actualities. From painful experience, I know that ideals don't always mesh with engineering realities at many times, which is funny coming from me who carries a soapbox around in case of opportunity.

[dan-golioth]

@Nummer378

So an allowlist removal implies exactly that: The opt-in becomes the default (and a opt-out never existed, the allowlist was always one-way only).

I can understand that perspective, but I think my broader point is that providing an opt-out when making the opt-in the default would have been a smoother migration. Once again, I understand that involves more maintenance burden in service of a seemingly marginal group of subscribers, and I understand there are always constraints. Even if an opt-out was not offered, I think more attention could be paid to curve compatibility (it doesn't seem that it was mentioned in any of the posts that have been referenced in this thread).

@Nummer378

I agree, a client that does not support P-384 would be in trouble. I was curious if you know of a particular implementation that fulfills these criteria, and if so what type of client that is (e.g. embedded TLS client? IoT? Why doesn't it support P-384? Space-constrained?).

Yes, on space constrained devices (e.g. microcontrollers), it is very possible that a small number of algorithms and curves will be supported with clients using TLS libraries such as mbedtls or similar. I have actively seen devices impacted by the change discussed in this thread.

[aarongable]

@dan-golioth

providing an opt-out when making the opt-in the default would have been a smoother migration

While reasonable in theory, in practice this stance in general requires service providers to continue providing both code-paths for twice as long. And folks complain just as vociferously when you remove the opt-out. So while I understand why you would want a "smoother migration", the reality is that no migration is smooth enough for everybody, and at some point you have to just make the change.

[mcpherrinm]

As a final point for this specifically: I would not expect us to make additional algorithm changes for the foreseeable future until post-quantum pki is sorted out. We will keep your feedback in mind for that transition.

[dan-golioth]

@aarongable @mcpherrinm thank you for your thoughtful feedback. I appreciate the work you do for Let's Encrypt and your transparency in this thread!