New Issuance Chains on Staging Failing

[aarongable]

(Pebble also does 100% asynchronous finalization, so folks doing client development for async finalization can test against that; they don't have to stand up a whole boulder instance.)

[imfranklin]

Thanks for the few-steps-back perspective. Our aim was to test the intermediate change, but as you point out our lack of stage certificates from a month or so ago leaves us without a proper test.

(A contact of ours from ISRG personally reached out to make sure we were aware of this change; that hasn't happened since the async finalization release which didn't go well for us, and so we were immediately very concerned...)

Without well-aged staging certificates and a somewhat involved ACME client replacement, we'll just accept that our hands are tied here. We have 30 day lookahead on certificate renewals, so should anything go awry June 6, we'll at least have the month to tend to whatever needs to be addressed.

[petercooperjr]

Yeah. From a testing perspective, there are many things changing, though well-configured systems shouldn't notice.

1. The intermediate changing, which has happened before but doesn't happen often.
2. The intermediate might be different for each renewal from that point on, so any manual steps that people have been in the habit of needing when the intermediate changes really need to get automated.
3. ECDSA leaf certificates will now always get signed by an ECDSA intermediate.
   • If one had gone out of one's way to get on the ECDSA-intermediate-allowlist earlier, this won't be new, but something else would change: The default chain for ECDSA leafs won't include ISRG Root X2 by default.
4. RSA leaf certificates won't have an "alternate" chain presented at all.

[imfranklin]

I think we'll definitely be using Pebble for testing as we migrate to certbot and a modern system overall, thanks! Appreciate this and all the other background info from the Let's Encrypt team here.