Temp support for X1? Lost access to large number of devices

From that file:

## Certificate data from Mozilla as of: Wed Apr 22 03:12:04 2015

Unfortunately, it looks like your ca-certificates bundle is 6 years old. This is from before ISRG Root X1 was even created, so ISRG Root X1 is not included. That's somewhat surprising because you said earlier that you have a fairly modern ca-certificates bundle installed:

At any rate, it sounds like your only option is to get a certificate from another CA. Sorry for the trouble!

4 Likes

Anyone know if GoDaddy has the same issue with openwrt or would it work as a quick and greedy fix? $93.00, yeesh.

1 Like

The best drop-in replacement for Let's Encrypt's services is usually another ACME CA, like BuyPass or ZeroSSL—you can potentially use the same tools to get certificates from these services, and they have at least some certificate products that are free of charge. But they have different trust roots compared to Let's Encrypt's.

Sorry, there's at least one other one in this category that I'm forgetting!

4 Likes

The problem is it would take days to go through the process wouldn't it? I can't recall anymore since I've been using letsencrypt for some time. It's fine, it's a temp fix and will give me time to sort out the problem.

I'll make sure to point out what happened, it's quite an oversight.

Thank you to everyone for all of the help.

2 Likes

The ACME certificate authorities are all automated so you should probably be able to get a certificate from them in just a few minutes, if you have sufficient administrative access to your server. This is part of the point of the ACME technology that Let's Encrypt developed. :slight_smile:

3 Likes

yup acme.sh with ZeroSSL works fine from my testing :slight_smile:

2 Likes

I went with GoDaddy since I know it well and it took all of 10 minutes to get the cert and another few minutes to install it. Things are back to normal finally. However, it's left a huge mess to figure out which I now need to document and figure out. I'm hoping I'll have a bit of help on this tomorrow.

Once things calm down, I'll spend some time looking into the alternatives mentioned here.

Thank you to all who helped, again, it was truly appreciated even if we didn't solve the problem.

3 Likes

What a mess, now I'm finding other servers that are having problems because they are using the letsencrypt certs on those servers too. Have to buy yet more godaddy certs.

2 Likes

I'm not sure, I understand you correctly, but if you are able to update your remote devices to trust the ISRG Root X1 certificate, you probably won't have to update the other servers.

2 Likes

He's in a catch-22 situation where he can't update the remote devices unless they can connect to the central server. But they can't connect to the central server until they get updated to trust ISRG Root X1...which is why the work around is to change CAs on the server to something that is already trusted by the devices.

5 Likes

It doesn't seem like it should have been catch-22, @jsha posted Help thread for DST Root CA X3 expiration (September 2021) on April 6th. To me it feel a bit like the theater is on fire :fire: and people are yelling fire :fire: yet not every got out of their seat until the flames :fire: actually reached them. The issue being the web world as a whole didn't take on all the duties and responsibilities to be ready for this expiration; the LE community did do due diligence on address and informing the powers that be.

3 Likes

As far as I understood the last message, he already got the devices back online with the godaddy cert at at least one server. Maybe I understood that wrong, and he is not able to update the devices through that channel ...

1 Like

Correct, remote devices stopped communicating as soon as the change was implemented with LE.
I found testing units and was able to confirm the problem and once I found out what it was, started asking how I might be able to add something back into the server, even temporarily.
Nothing worked so I went with a GD SSL cert. The moment I restarted the web server using GD certs, all remotes started communicating.

Someone will have to build updated versions for the devices but now I'm reading in the openwrt forums that only source code users will be able to use curl with updated SSL while those using image builder will be SOL unless the curl maintainer updates.

I'm not sure, just reading as I look into this now that things are a little calmer.

And yes, it's easy to blame that someone overlooked something this important but there is a lot to technology and some things can fall between the cracks. Usually not this big but even the largest companies make mistakes and learn from each one.

2 Likes

@Bruce5051 that's not how I'd put it. Maybe more like we picked up the Internet and shook it around and found a few loose bolts here and there, which we're now helping people tighten. In an ideal world everyone would get our announcements about upcoming changes and know exactly how to test them, but this type of change (expiration on a future date) is quite hard to test accurately in advance.

That is probably true for some users, but your issue won't be solved by updating curl or SSL alone. You also need to get an updated ca-certificates bundle onto all your devices.

4 Likes

Yes, indeed, that will be updated in the packages on the new firmware.

I know so far that someone did see the notice about the upcoming change but there was also a mention that for the most part, everything should keep working so it didn't seem like anything all that urgent. No one took into account the openwrt software not being well updated or even mentioned for this scenario.

From what I see in the openwrt forums, and I could be wrong, the developers are basically taunting those who don't know how to use source that it's their own fault.

Not everyone has the same skill sets. Some use source, others don't but get by just fine except for that one time where everything breaks. Projects need to understand that not everyone that uses their solution is a high level techy that has time to learn everything about everything. Most of us know a little about a lot of things and leave the rest to pro's when possible.

I won't mention whom but in the initial answers to my own question, I felt like I was being chastised for not knowing everything about SSL and the person just kept on going with high level information that folks like me will not understand because SSL is a small part of our lives. We configure machines, generate certs, install them and we don't need to learn everything about SSL.

I'm happy this was a temporary problem and love that organizations like LE exist. There are a lot of things that should not cost people on the Internet and basic security that slows hackers down is one of them.

I'll keep using LE and recommending it.

7 Likes

I'm sorry to hear that. That sort of experience - chastising someone for not knowing enough - is all too common in tech forums, and it's something we actively try to avoid here. Sounds like we didn't do a great job of that today, but we'll try to do better in the future.

5 Likes

Everyone was amazing. Only one person sounded that way and I just looked past it because as you said, there are always those types in forums.

3 Likes

I have a few OpenWrt devices that the maintainers and developers will not update OpenSSL to 1.1.1 branch. Basically since OpenSSL 1.0.2 is no longer supported then neither is my product. But OpenWrt aren't the only ones either for example my QNAP NAS.

uname -a
Linux NAS3BA281 4.2.8 #2 SMP Thu Sep 23 06:02:16 CST 2021 armv7l unknown
[~] # openssl version
OpenSSL 1.0.2za  24 Aug 2021

I am not chastising individuals in my comment above, I was trying to say if the community had taken this more seriously the impact could have been reduced. I apologize for being rough and / or harsh.

3 Likes

It wasn't you I was talking about but all good :).

2 Likes

Should I ask?
Was I not myself during that extended sleep/beer/chocolate/ice cream withdrawal experience?

Was it me?

2 Likes