To minimize bandwidth (which is incredibly expensive for our product), some of our client apps bundle intermediate certs with them. They were previously only bundling E1, but we have now rolled out a new version including E5-9. However some of the older versions still only bundle E1. Is it possible to continue issuing certificates via E1 until that intermediate is retired? I had hoped that --preferred-chain E1 would do this, however it reports "Certbot has been configured to prefer certificate chains with issuer 'E1', but no chain from the CA matched this issuer. Using the default certificate chain instead." and I think this --preferred-chain is only working for root rather than intermediate certs anyway?
No, preferred issuer is only used to specify roots of cross-signed chains (where there may be more than one option), but not intermediates. It's not used when ordering your certificate, so the issuance has already happened and the ACME protocol doesn't have a concept of asking for a particular issuer to be used for the actual issuance.
You should generally not need to distribute intermediates to trust stores, instead you should ship a list of trusted roots and implicitly trust any valid intermediate issued by those roots, as the CA can switch to new intermediates at short notice. Services your clients connect to will serve these intermediates as part of their certificate chain.
We (the community) have seen in particular many IoT projects are bundling and trusting specific intermediates and this has caused them serious outages and problems with updating their devices in the field. I don't know if you product is IoT or something else.
7 Likes
Btw does your app trust isrg root x2? It feels like it does on old thread and because it's using ec key it'd much smaller than rsa signiture
6 Likes
Yes we do X2 / ECDSA so there is already some bandwidth saving.
I do appreciate that you need to be very careful about monitoring and keeping update with the intermediate certs, however it does save us a significant amount of money (because we are paying for expensive MNO zero-rated data for millions of users) if we can manage to make this work correctly by not needing to send intermediate certs on every connection. We've managed these intermediate rotations fine in terms of keeping the stores up to date with all the root+intermediates, but it would be great if there was some way to have a similar nudge like --preferred-chain for allowing us to provision on the older intermediate certificates for a longer time. If not, we'll have to find a way to out-of-band update the certificate store on the devices rather than doing it baked in to the apk in this case and forcing users to upgrade a few months before the intermediates are set to expire.
It sounds from the responses above and the note on one of the pages I saw about potentially rotating intermediates every year that this is cutting against the grain of the way LE is heading, so in this case I guess we have to figure out out-of-band certstore updates or another way of running this somewhat unusual setup.
Well, just a nit, pinning intermediates has always been frowned upon 
It was always possible an intermediate could get revoked and a backup intermediate used at any time. One benefit of the new style of varying intermediates for each issuance was to expose these cases.
Just in case you have not seen it yet you should study the below page. Especially the description of -preferred-chain to the X2 root and the Backup section for Intermediates.
5 Likes
The E1 intermediate is retired and won't be issuing any more certificates.
It sounds like what you're trying to do is specialized enough that you'd be better off using your own private PKI rather than trying to integrate Let's Encrypt's publicly trusted certificates. The public WebPKI is really designed around each server sending the intermediate on each connection, with the clients only having the roots, and not caring about which intermediate is being sent.
7 Likes
Thanks, I had forgotten we had previously considered this, but indeed it does sound like running our own CA for this particular domain may be the most sensible approach - I think we were trying to ensure easy compatibility with the rest of the internet, but perhaps we should revisit breaking that to enable the data efficiencies that are required.
5 Likes
And if you really need to squeeze out every last byte, using your own P-256 root for signing the cert you're using will give you slightly smaller signatures than the P-384 intermediates that Let's Encrypt uses (assuming that's a good enough level of security for your application).
7 Likes
Adding to this, ISRG/LE have indicated they do not want users to be able to select which active intermediate is used.
The advice from @petercooperjr about migrating to a private CA is spot-on. Depending on how your apps/devices work, you might be able to use a custom trust store that is just your root added to the system or package certificates bundle.
6 Likes
I understand, however if your clients will fail en-masse if the wrong intermediate I can imagine some regret saving money on not sending intermediates. As you clearly control the stack at a sophisticated level you could download unknown intermediates (and even roots) on demand from your own services, Windows has a similar-ish thing that happens when you visit sites (intermediates from trusted roots are stored) or get windows updates (trusted root certs are updated from a controlled set).
6 Likes
You should get interested in RFC 8879: TLS Certificate Compression.
There should be few implementations available already, and since you control both clients and servers, I think that would be the best way forward.
OpenSSL >= 3.2 supports this, possibly others. RFC has been co-created by Google and Cloudflare, I'd assume you have high chance seeing them implemented in their services.
6 Likes
Sidenote:
If you are not doing this already, many servers and gateways now offer hooks into the TLS protocol. If I were trying to maximize bandwidth, I would look into using the Client Info during the TLS handshake to determine which certificates and chains to use. You can often make some educated guesses about the client based on that information (which is the supported tls versions and ciphers).
Some servers now natively support simplified versions of this. For example, Nginx recently supported a "hybrid" setup where you configure an ECDSA certificate for primary use, but will fallback to a RSA if the client does not support it.
7 Likes
Thanks everyone for the ideas here. Going to own CA seems to be the most sensible solution - TLS certificate compression certainly seems nice but not really implemented widely enough at present for us to do this unfortunately.
Given we're going own CA route then auto-detection of client etc doesn't seem to make a lot of sense - we'll just have a standard port using letsencrypt with chain, and a different port using our own self-CA which the client will connect to in preference, if TLS error (eg old client) it will flip to the letsencrypt one.
5 Likes