I'm not an expert anything SSL, so excuse me if I'm not using the correct terminology. But I think we accidentally did certificate pinning!
My place of work have some services running in an Azure Kubernetes services, and we use CertManager to automatically supply SSL for these. One of these services is an mqtt broker that a bunch of IOT sensors use send telemetry.
Today that certificate was automatically renewed by the Cert manager (successfully), but the new certificate is signed with the R11 intermediate certificate. However, it turns out that our sensors had been pinned to ISRG Root X1 + R3, and will no longer connect.
R3 is still valid for about a month, which is plenty of time for us to fix the issue... IF we can get the devices online again.
So my question is: Is it possible to reissue our certificate, signed with R3, rather than R10/R11?
If not, our only solution may be to manually visit every affected piece of hardware and manually update them. They are scattered across my country...
Uh, no. Having anything like that pinned to an intermediate doesn't make any sense. The intermediates can, will, and do change regularly. Sometimes, as in this case, with plenty of notice. But Let's Encrypt also keeps intermediates in reserve for disaster recovery, so they might have switched to R4 at any point in the past few years without any further notice.
If you have the certificate and private key from before your most recent renewal, you might be able to roll back to that to use it temporarily if it's not yet expired.
Your old cert for gateway.remoni.com (issued by R3) is still valid until 2024-07-17T12:32:48. You can manually install this certificate (you're reusing the private key, so you still have access to that) to give you 30 more days to fix this issue.
PS: For reference, here's your new certificate (issued today; by R10):
All publicly-trusted certificates are logged, primarily so that people can confirm that the Certificate Authorities are doing their job correctly, but also so that site owners can learn about all certificates that have been issued for their domains.
But I wouldn't treat it as a "backup" on its own.
Make sure your update changes your devices to only have roots in the trust store. Intermediates should be sent by the server as part of the TLS connection, the client should only care about the root. And for devices like what it sounds like you're using, you probably want more than one root in its trust store, perhaps including one you create yourselves. Let's Encrypt is great, but if their system goes down for an extended time or they need to change their root certificate due to a compliance issue or whatnot, you want the flexibility to be able to switch to another CA if needed.
You can find an overview of the currently known free ACME CAs here:
BuyPass is a good no-nonsense ACME endpoint (no EAB et c.), but as far as I know only has an RSA root. But that's not really a big deal if it's a backup CA, unless you really don't have the bytes to spare. That said, you're now removing an intermediate cert from the FW, so you have at least some room
I used certificate transparency logs. crt.sh is the most convient/easy to use one, but there are also others. For example earlier when I looked up your certs crt.sh was down, so I used search.censys.io.
@MikalJ remind your developers that they should be trusting the root certificate ISRG Root X1 (and optionally ISRG Root X2), plus they should trust other root certificates from common CAs so you can change CA at any time.
They should not use a local store of intermediates as the trust store, instead they should probably trust any valid (non-revoked, non-expired) intermediate issued from a trusted root (and any level of intermediate signed by a valid intermediate etc) - that's generally how certificate trust works best with minimal intervention. You have to allow for the CA to suddenly change intermediate and you have to allow for roots eventually requiring updates/changes.
There are some great points here that we will take to heart. I really appreciate all the feedback, and would like to thank you all for giving your two cents.
I apologize for reviving an old thread. I came across this issue online and encountered the same difficulty. My new certificate private key has changed, and I can't find the old one. Is there any other way to get my R3 middleware device online? I've tried looking in the Caddy data folder and logs and even attempted to use disk recovery tools but with no luck.
@Seunji why are you trying to recover an old certificate? What's wrong with using a new certificate? You can also get Caddy specific help at https://caddy.community/
Thank you for your response!
My IoT device's firmware has the root certificate fixed to the R3 intermediate certificate, and now it's unable to connect to the broker, just like the original poster faced. I will also check out the Caddy forum.
Thank you very much!
If you can't find the private key of the original certificate (the cert itself can be downloaded from https://crt.sh), there's not much we can do. Issuance by R3 has stopped and that intermediate cert has been retired and it's not possible to get another cert issued by it.
