We have a number of iot devices around the world With a certificate issued by ISRG Root X1 and issued to R3 (valid until sept 2025). From what I've read in other threads, this is a less than optimal thing to do, but here we are, trusting an external embedded firmware consultant.
Due to a recent automatic renewal of the certificate for our back end API (IIS 10/Windows Server 2022, using win-acme), all the devices stopped working a few days ago.
My question is basically if there is any possibility to create a temporary certificate for our IIS API that will allow the iot devices to connect with https so we can flash new firmware images? If not we'd have to issue a recall of the devices.
If your embedded devices require a certificate issued by R3, then no: it's not possible any longer to get a certificate issued by that intermediate certificate.
Hopefully you pinned some backups.
Also, although I think you've already realised this: never pin to intermediate certificates.
Thank you Osiris! I feared that was the response. And yes, we will likely avoid depending on https and thus certificates for firmware OTA in future versions.
No, you should depend correctly on HTTPS. By not pinning intermediate certificates like R3, but root certificates like ISRG Root X1/X2. And always add a backup root certificate (although that might be tricky if storage space is really sparse).
Relying on HTTP instead of HTTPS is a bad idea security wise. (Unless you have some other secure method of validating the firmware like with a secure signature.)
I agree. But still I am thinking that if TLS fails, we can use a fallback with a unique device key and AES-GCM encryption both ways to do the firmware OTA upgrades only.
You are getting into disaster territory, don't invent new ways to deliver OTAs. Whatever you do just use something that has been properly vetted and battle tested, Daily billons of devices running android are updated, check what they use and adopt that to your custom solution.
Your approach of assuming things about TLS clearly backfired. There is no way to issue certs from retired Root CA.
It's not actually the root CA that has been retired, it's just the intermediates which were changed over, which is something that could happen at any time and thus should not be relied upon to begin with. The roots are fine. (For now, as they will also need to be replaced somewhere down the future.)
With iot devices that might be offline for some period of time before they reconnect, the whole concept of relying on regular firmware updates with updated certificates is a bit flakey imo.
But, the only thing that might change are the root CA certs. These are very long-lived. The leaf and intermediates you get from your TLS Server each time you connect. You want to validate that the leaf and the intermediate chain leads to a trusted CA root you already have.
And, for resilience against disaster you should have a couple / few CA roots that you trust.
As an aside you may find the old cert is still in the Local Machine certificate store (either under Web Hosting or My) and you could manually edit your https binding to select that cert again. You could also have a look under C:\ProgramData\win-acme\acme-v02.api.letsencrypt.org\Certificates for the original PFX.
Generally the right thing to do is trust the roots (for multiple CAs so you have a fallback), and by extension any intermediate signed by a trusted root, but it seems a few other IoT developers have followed the same path of just trusting the intermediate (which can of course change overnight).
Maybe you should try talking to the higher ups at ISRG. Since the R3 is still valid, there really should be a way to get another R3 certificate, especially if you can pay for it.
In general you should be aware that you can be held liable for the devices you sell and therefore should know how they work and what they depend on before selling them to end users. Let's Encrypt informed months ago that they're going to remove R3 from the chain.
But if you get this wrong you will be in even deeper trouble. You will have to sign the firmware and do all sorts of checks to ensure the data isn't tampered with. The lesson to learn here is to not rely on third parties if you don't have to. In this case, publicly trusted CAs. For your own devices you can just as well use standard TLS with your own root certificate to trust and pin. You don't need domain validation or anything like that. As long as you keep the private keys save, you can just issue your own leaf certificates that your devices will trust.
While I do not officially speak for Let's Encrypt, I strongly doubt they will spin up an entire production line just to issue a leaf certificate. I will contact the staff directly regarding this matter though for them to comment officially, if they wish, for the sake of being able to know and reference their stance going forward, if nothing else.