Temp support for X1? Lost access to large number of devices

We have a large number of remote devices which have stopped communicating. Since they are not communicating, we cannot send them any updates.

curl: (51) Cert verify failed: BADCERT_EXPIRED

Our only option would be to either find a cert provider that still supports X1 or find a way to add X1 support on our server.
Is there a way to add support for X1 on the server just long enough to recover the devices?
I don't know enough about SSL to know how I could do this. Perhaps an intermediate file with the other letsencrypt files that the apache server is using?

1 Like

What X1 are you talking about?

ISRG Root X1 is still alive and well. All certs currently issued chain up to this root, this has not changed. What has changed is that DST Root CA X3 is now expired.

(There used to be an intermediate certificate called " Let's Encrypt Authority X1", but that was retired in 2016 - 5 years ago, so you're surely not talking about that)

5 Likes

Your server probably sends the incorrect chain. However, as you didn't fill out the questionnaire which should have been presented to you when you opened this thread, we can't help you with details, just a generic hint: reconfigure your webserver so it sends the correct certificate chain.

3 Likes

Everything stopped working just hours ago without us making any changes to anything.
The dev I spoke with said it was because X1 was no longer supported so I don't know, I'm caught in the middle.

I was not able to fill out the info asked for because it is not my network so cannot post that info on a public forum.

just a generic hint: reconfigure your webserver so it sends the correct certificate chain.

Maybe that's what I'm asking about. Can you shed any light on this, url, article, something that explains a little more.

1 Like

I'm sorry to inform you that your dev doesn't know what he/she's talking about. Or there was some kind of miscommunication between the two of you, which is also likely. ISRG Root X1 is a public/private keypair and is alive 'n kickin'! It's just that the default certificate chain includes a cross-signed certificate for the ISRG Root X1 keypair which was signed by a now expired DST Root CA X3.

This really depends on your infrastructure. It's impossible to write an article or how-to for thousands of different pieces of software using TLS certificates, so I'm afraid something like that is not available.

I can say two things which might be applicable with possible solutions. Your dev should know what it means, although I'm actually doubting that a little bit looking at the X1 statement earlier..:

  • If your TLS service is not sending any intermediate chain, it should. There currently are two chains available. See Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt for more info.
  • If your TLS service does send intermediate certificates and it's using the currently default certificate chain (which includes the cross-signed ISRG Root X1 which was cross-signed by the now expired DST Root CA X3), it might help to change from the default chain which includes the aformentioned cross-signed ISRG Root X1 to the certificate chain with just the R3 intermediate.
3 Likes

It is likely myself that misunderstood and I did point that out. I'm barely familiar with SSL other than installing it on servers etc. All I know is that remote devices stopped communicating some 6-7 hours ago and are all stuck. They are all remote so there is no way to recover them other than having them all shipped back. The only way to update them is to get them to communicate again which is why I was asking if there is some way to go back in time, to have some time to send them an update.
Going back in time meaning, supporting what ever was changed for a little while.

The server can do what ever I want it to, I just need to know what I need to do to make things work again since the person that usually does this is incapacitated and I'm trying to help.

The server is a centos-8 running apache 2.4 if that helps.

2 Likes

Knowing you're using Apache partly helps. It uses the SSLCertificateFile (and for older versions also SSLCertificateChainFile to load the certificates and its chain. Depending on what the value of that/those directives is, we can look further. Please also state the exact version of your Apache, as that matters too.

2 Likes

Sure...
httpd 2.4.37 running on centos -8/

In the httpd configuration;

#Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/chain.pem

% ll /etc/letsencrypt/live/domain.com/
cert.pem -> ../../archive/domain.com/cert1.pem
chain.pem -> ../../archive/domain.com/chain1.pem
fullchain.pem -> ../../archive/domain.com/fullchain1.pem
privkey.pem -> ../../archive/domain.com/privkey1.pem

I notice that on other servers, this is a bit different.

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/domain.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem

1 Like

What type of devices are your remote devices?

They probably don't have the ISRG Root X1 cert in their trusted roots. Up till today, this was not a big problem, because that ISRG Root X1 cert is cross signed by the DST Root X3 certificate which provided a valid trust anchor. But since the DST Root X3 expired today, most of the devices that rely on it as trust anchor begin to reject the certificates (an exception are older Android devices, which don't check the valid time of a trusted root)

So I see the following possibilities to get your devices back online

  1. If they have an root certificates other than DST Root X3 installed (maybe some other certificate provider, ...) you can (at least temporarily) install a certificate from such a provider on your server and then update your devices to support ISRG Root X1 as trust anchor

  2. Find any other way to install ISRG Root X1 as trust anchor (maybe even manually ...)

3 Likes

Mostly IoT devices out in the field. The ones that stopped working are running openwrt versions with an older 1.0 openssl. There is no way to update them unless I can give them some way to communicate again, even temporarily as you said.

Does anyone know of an SSL provider that would still provide the old method?

Does anyone know how I can install this ISRG Root X1 trust anchor? I can do the work such as changing anything on the server or adding a file, etc, but am not familiar enough with SSL to know most of what you are suggesting.

1 Like

If you're currently sending the default chain and it doesn't work, it might help to use the alternative chain with just the R3 intermediate cert, i.e. without the cross-signed ISRG Root X1 cert.

But that would require having the ISRG Root X1 cert in their trust stores.

1 Like

So pretty much, I'm SOL I guess.
There is no way to recover the devices by doing something on the server side.
I hoped maybe there was something I could do on the server, adding temporary support to allow the devices to communicate long enough to update them.

What a mess. Times are hard enough these days, this is just extra painful stress now.

1 Like

The ISRG Root X1 has to be installed on the device and not on the server, but it seems that's not an option.

And without knowing which alternate trusted roots are actually installed on the device, it's quite pointless to suggest any provider. Oftentimes IoT devices are very stripped down to the bare minimum, so that even most of the default root certificates are removed.

And also openssl 1.0 sort of rings a bell, because if I recall correctly, this version will always walk up the whole chain (and therefore fail at the expired DST Root X3), whereas openssl 1.1 will accept the chain once it finds a certificate installed as trusted root (and therefore accept at ISRG Root X1, if it is installed at the device)

2 Likes

That's true. But to be honest, this whole scenario has been anounced for at least half a year now. This should have been plenty of time to test and update your devices accordingly ...

2 Likes

Yes, if the right people would have been at the job. Sometimes, things are overlooked and this was a huge one. All i know is that those devices are running openwrt ca-bundle - 20200601-1 and I can't find detailed information to know what I might be able to do on the server side.

I appreciate the help. Thank you.

1 Like

This might indeed contain the ISRG Root X1 certificate, but as your openssl version is quite old, it still fails (see DST Root CA X3 Expiration (September 2021) - Let's Encrypt)

But there may be hope, also like @Osiris mentionend:

Currently you are probably serving the default chain, which also contains the now expired DST Root X3 certificate. Openssl 1.0 can't deal with that. But you can (at least temporarily) switch to the alternative chain (on the server!). This alternative doesn't contain the DST Root X3, and therefore even the older openssl 1.0 should be able to deal with that.

I don't know, how you create certificates on your server (and from your your posts I suspect it's not your personal responsability). So you should get the person who is responsible to create/renew those certificates and see if he can switch to the alternative chain, then your devices probably will come online again ...

And as your devices include a default ca-bundle, it might as well be possible to just serve a certificate from any other provider than LE, because the most common ones should be able to be verified ...

2 Likes

As luck would have it, I'm the only one here as this happens.

The server has certbot on it and I did renew the certs using that earlier today thinking maybe they were simply expired. Bad enough but simpler problem :).

So long as I know what I'm supposed to do, I can do it.

I assume certbot is using the default chain and I posted that info above.

cert.pem -> ../../archive/domain.com/cert1.pem
chain.pem -> ../../archive/domain.com/chain1.pem
fullchain.pem -> ../../archive/domain.com/fullchain1.pem
privkey.pem -> ../../archive/domain.com/privkey1.pem

Is there some info online on where I can get the alternative file then use it temporarily?

1 Like

I should clarify that I am a server admin so very familiar with Linux. I just don't know the in and out's of SSL certificates other than installing them, LAMP, etc. With some guidance or info on what I need to do, I'm sure I can get it done.

1 Like

Probably you can buy a cert from some cert authority who's in your devices' CA store and use it on your server?

2 Likes

Yes, but this sounds much more hopeful if I can find more information on how to accomplish this.