This might be wishful thinking and/or crazy talk, but were these devices set up so that they synchronize their time from a remote NTP server that you control? Or can you trick their DNS resolvers into using a different NTP server? If so, you could perhaps set the time back for them all, and then communicate with them again to install the new ISRG Root X1 certificate, and then put the time right again. Of course, that might have adverse effects.
It's a pity (and fairly surprising) that you don't have SSH access. I imagine it's a fairly normal way to communicate with OpenWRT devices, and it would be unaffected by this problem.
Maybe not so much wishful thinking but a lot of creative ideas came up on that day, trying to figure out how to regain control of them.
SSH is available on them but only if someone put it directly onto the Internet with a forwarding port. Otherwise, there are no services accessible on them which is specifically how they were built so that they are safer, more secure for the end location.
They do use NTP but ntp.org.
DNS, there would have been no way to do anything there since we could not send them any controls to do anything because they were simply not communicating as of the LE change.
They function perfectly but in this case, this change was not caught in time or overlooked and this happened.
The question for me at this point is why did all this happen to begin with? While it was a nightmare, the solution was to buy a GoDaddy cert and all went back to normal. Why didn't LE support what ever GD is doing so that this would not have happened?
While this was happening, I came across countless posts across many sites from people who were experiencing all kinds of problems too so we were not alone.
This happened because root certs need to expire at some point. The root certificate that expired claimed to be issued in the year 2000 - that's older than pretty much every computer I currently use. It is unwisely to use such old things forever, because the security of the entire PKI/web depends on them. They need to be removed at some point, and September 30 2021 was this day.
Other CA's like GoDaddy simply use other root certificates, that don't expire this year - but they will some day. Maybe in 2022, maybe in 2030, maybe in 2040 - but they will at some point.
Since root certs expire, new root certs are issued to replace them. However, if systems do not get updates, they won't trust the new roots - now what do you do? This will inevitably break at some point, until the industry learns that not updating internet connected devices for 5+ years is not an option.
Let's Encrypts root certificate was accepted into root store programs roughly in 2016 - 5 years for software updates to distribute this certificate. For 5 years some systems were not updated, so it broke now.
In addition to that, Let's Encrypt decided to still serve a chain up to this (now expired) root certificate delibaretly. This fixed older Androids (those not having a new enough trust store - the update story again), but broke libraries with incorrect chain verification logic. This was a calculated risk Let's Encrypt decided to take, to help Android users. This issue can be mitigated by serving Let's Encrypts alternate chain, but it won't help systems without an updated trust store.
Looking at this thread, your issue apparently was that while you had the intention of serving reasonably up to date software, you actually weren't. These type of supply chain issues are hard to come by, I know that. I also want to point out that I don't intend to blame you or your company in any way. Rather I blame poor industry standards. You're not alone with these issues. I own a Smart TV by a well-known company that hasn't gotten an update to its root store since about 2012-2013, even though it is much more recent and has gotten software updates since then. Needlessly to say that it won't work with any LE site as of a certain day.
I fully agree, security should be upgraded on a regular basis and in our case, we didn't take into account that openwrt would not be keeping non source based packages updated which lead to this. Even the newest version doesn't work unless you know how to use source.
To me, that is not a responsible way to offer a project and any project worth its salt should have dealt with this for its masses. I'm sure that anyone who develops with openwrt will not agree but they need to keep in mind that in all projects, there are different levels of knowledge.
BTW, the reason I believe that our devices were overlooked is because they aren't directly connected to the Internet, they are merely clients on the LAN and really not any kind of security issue. They only need to communicate outward so no one was thinking about something like this.
I imagine that a large number of companies with embedded devices across the Internet will have been hugely affected by this.
And now I better understand about GD too, it's only a matter of time. In that case, I hope openwrt updates all of the packages that use openssl without forcing people to have to change out countless devices out in the field.
it's outside of this forum's subject but what's your device? I have openwrt buildroot installed for my potato in pocket so maybe I'm able to compile something for you if you want - if you trust me for compiled image/ipk file