Temp support for X1? Lost access to large number of devices

Yes, exactly what I thought when reading this thread, any cheap cert will do temporarily -- if supported by the devices. This doesn't have to be solved LE-wise, @Lorance can just switch back to LE when the IOT devices are upgraded.

I agree but it sounded like there is a way to use an alternate chain that could do the trick even faster than having to go through the long process of getting another SSL cert from another provider.

There are other free ACME based certificate authorities you could use via certbot as well. ZeroSSL is probably closest to LE in terms of features and capabilities, but requires setting up an account on their website and generating "External Account Binding" (EAB) credentials first. BuyPass can be simpler to setup because there's no EAB necessary. But they don't support wildcard certs and you can only have up to 5 names in the SAN field.

  1. Check that your server software either is configured to use fullchain.pem OR (but usually this is the wrong way unless the server software is ancient) both cert.pem and chain.pem. Figure out which is the case. If neither, that's a different mistake - let us know.

  2. To change what your server sends to clients to omit mentioning the expired DST Root CA X3, you would entirely remove the last section of the fullchain.pem (or chain.pem in that second case where you're relying on that). You should see it has several sections that look something like


Note that scrolls, at least in my preview, so check how the whole thing looks. Make a backup of the file you're changing just in case, put it to one side. You want to remove only the LAST one of these sections, in its entirety. If I correctly understood your situation, this fixes the problem. Once you've made the change, restart the server software (if you somehow have no idea how to do that, a very drastic approach is to reboot).


In this servers case, it's set like this, no fullchain.

#Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/chain.pem

Ok, so you want me to edit out the second part of the cert then restart the httpd service.
I see this in both chain1.pem and fullchain1.pem. Should I edit both?


Yep, that's correct. And great writeup, @tialaramex. Thanks!


I backed everything up and edited both and restarted the web services.
As you can see, only the chain.pem is being used but I thought I'd do both since it cannot hurt much.

No change.

Cert verify failed: BADCERT_NOT_TRUSTED

1 Like

Where are you seeing this error? And in response to what command?

I'm logged into a device on my desk so I can test.
The command that the devices are using is;

% curl --cacert /etc/ssl/certs/ca-certificates.crt https://www.domain.com/app.php
curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
% curl https://www.domain.com/
curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED

Same no matter if I use the app url or just the domain.

However, I get no error when doing the same to google, facebook, etc.


Could you show me the output of this command on the device:

curl --version


grep cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 /etc/ssl/certs/ca-certificates.crt (this checks whether your ca-certificates bundle includes ISRG Root X1, by search for a line from the PEM encoded certificate at https://letsencrypt.org/certs/isrgrootx1.pem).

1 Like

This device is a really old one. I'm trying to fire up a newer version.
However, even this old one was working up until this morning.

% curl --version
curl 7.40.0 (mips-openwrt-linux-gnu) libcurl/7.40.0 PolarSSL/1.3.14
Protocols: file ftp ftps http https
Features: IPv6 Largefile SSL

% grep cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 /etc/ssl/certs/ca-certificates.crt

Nothing returned.

Note that I also tried with a 21.02.0 build earlier and got a different error,
curl: (77) CA signer not available for verification

I've not made any changes on the server other than the certs test I was asked to do above. Things just stopped working and it's been like this since.

1 Like

The curl on your device is using PolarSSL rather than OpenSSL as we've been assuming. I'm guessing PolarSSL has similar issues to OpenSSL, but I'm not sure the proposed workaround will work. I don't know anything in particular about PolarSSL.

The output of the grep command suggests your device doesn't have the ISRG Root X1 certificate at all, which would mean there is no easy fix other than going into the field to update the devices. However, you mentioned this is an old device. Can you run the same grep command on a more up-to-date device?


Yes, I'm trying to get a hold of a newer one now and will run the same tests once I have one running.

1 Like

While you're work on that, can you also post your actual domain name? We'll need that in order to verify the actual certificate chain being served.

1 Like

I cannot post that, it's not mine to share. I can get what ever info you need using something like SSL Checker if you want? Which part do you need?

1 Like

download.openwrt.org (openwrt source/package repo uses LE too, but they chooes to use short chain. maybe it need short chain?

1 Like
Chain Information
Certificate # 1 - Common Name: domain.com
Decode this certificate for verbose information →
Subject Common Name 	domain.com
Issuer Common Name 	R3
Issuer Organization 	Let's Encrypt
Not Before: 	Sep 30, 2021 16:12:06 GMT
Not After: 	Dec 29, 2021 16:12:05 GMT
Signature Algorithm: 	sha256WithRSAEncryption
Serial Number: 	xxx
SHA1 Fingerprint: 	27:A6:46
MD5 Fingerprint: 	C4:EF:39
Certificate # 2 - Common Name: R3
Decode this certificate for verbose information →
In place? 	Yes, this certificate directly certifies the preceding one
Subject Common Name 	R3
Subject Organization 	Let's Encrypt
Issuer Common Name 	ISRG Root X1
Issuer Organization 	Internet Security Research Group
Not Before: 	Sep 04, 2020 00:00:00 GMT
Not After: 	Sep 15, 2025 16:00:00 GMT
Signature Algorithm: 	sha256WithRSAEncryption
Serial Number: 	xxx
SHA1 Fingerprint: 	48:78:2C
MD5 Fingerprint: 	5D:7C:43
OpenSSL Handshake

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain.com
verify return:1
OCSP response: no response sent
Certificate chain
 0 s:/CN=domain.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
Server certificate
issuer=/C=US/O=Let's Encrypt/CN=R3
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: X25519, 253 bits
SSL handshake has read 3316 bytes and written 302 bytes
Verification: OK
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: xxx
    Master-Key: 6FF36...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:

    Start Time: 1633047903
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

Does this help at all?

1 Like

Can you post what is in that /etc/ssl/certs/ca-certificates.crt file on the device(s)? Or at least its file size, or the number of times "BEGIN CERTIFICATE" appears in it? What I'm afraid of is that it may only have the expired cert in it, in which case I don't know what you'd do. But if it has a lot of roots in it, even if ISRG Root X1 isn't in there then you may have some options for other CAs.

1 Like

Ok, here is the newer version, the majority of devices that we have running.

% curl --version
curl 7.60.0 (mipsel-openwrt-linux-gnu) libcurl/7.60.0 mbedTLS/2.16.8
Release-Date: 2018-05-16
Protocols: file ftp ftps http https
Features: IPv6 Largefile SSL

% grep cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 /etc/ssl/certs/ca-certificates.crt

No results.

% ls -la /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 258424 Jul 8 2019 /etc/ssl/certs/ca-certificates.crt

certs.txt (256.3 KB)


The confusing part to me is all our sites are using letsencrypt certs and all of those are broken when using curl on openwrt. However, others like google, facebook, pinterest, none of those are broken. I also tested every same site using curl on Linux and Windows and those are all working. Only openwrt curl is not working, or at least, all I know about right now.

1 Like