Temp support for X1? Lost access to large number of devices

Yes, exactly what I thought when reading this thread, any cheap cert will do temporarily -- if supported by the devices. This doesn't have to be solved LE-wise, @Lorance can just switch back to LE when the IOT devices are upgraded.

I agree but it sounded like there is a way to use an alternate chain that could do the trick even faster than having to go through the long process of getting another SSL cert from another provider.

There are other free ACME based certificate authorities you could use via certbot as well. ZeroSSL is probably closest to LE in terms of features and capabilities, but requires setting up an account on their website and generating "External Account Binding" (EAB) credentials first. BuyPass can be simpler to setup because there's no EAB necessary. But they don't support wildcard certs and you can only have up to 5 names in the SAN field.

5 Likes
  1. Check that your server software either is configured to use fullchain.pem OR (but usually this is the wrong way unless the server software is ancient) both cert.pem and chain.pem. Figure out which is the case. If neither, that's a different mistake - let us know.

  2. To change what your server sends to clients to omit mentioning the expired DST Root CA X3, you would entirely remove the last section of the fullchain.pem (or chain.pem in that second case where you're relying on that). You should see it has several sections that look something like

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

Note that scrolls, at least in my preview, so check how the whole thing looks. Make a backup of the file you're changing just in case, put it to one side. You want to remove only the LAST one of these sections, in its entirety. If I correctly understood your situation, this fixes the problem. Once you've made the change, restart the server software (if you somehow have no idea how to do that, a very drastic approach is to reboot).

6 Likes

In this servers case, it's set like this, no fullchain.

#Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/chain.pem

Ok, so you want me to edit out the second part of the cert then restart the httpd service.
I see this in both chain1.pem and fullchain1.pem. Should I edit both?

2 Likes

Yep, that's correct. And great writeup, @tialaramex. Thanks!

5 Likes

I backed everything up and edited both and restarted the web services.
As you can see, only the chain.pem is being used but I thought I'd do both since it cannot hurt much.

No change.

Cert verify failed: BADCERT_NOT_TRUSTED

1 Like

Where are you seeing this error? And in response to what command?

I'm logged into a device on my desk so I can test.
The command that the devices are using is;

% curl --cacert /etc/ssl/certs/ca-certificates.crt https://www.domain.com/app.php
curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED
% curl https://www.domain.com/
curl: (51) Cert verify failed: BADCERT_NOT_TRUSTED

Same no matter if I use the app url or just the domain.

However, I get no error when doing the same to google, facebook, etc.

2 Likes

Could you show me the output of this command on the device:

curl --version

Also:

grep cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 /etc/ssl/certs/ca-certificates.crt (this checks whether your ca-certificates bundle includes ISRG Root X1, by search for a line from the PEM encoded certificate at https://letsencrypt.org/certs/isrgrootx1.pem).

1 Like

This device is a really old one. I'm trying to fire up a newer version.
However, even this old one was working up until this morning.

% curl --version
curl 7.40.0 (mips-openwrt-linux-gnu) libcurl/7.40.0 PolarSSL/1.3.14
Protocols: file ftp ftps http https
Features: IPv6 Largefile SSL

% grep cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 /etc/ssl/certs/ca-certificates.crt

Nothing returned.

Note that I also tried with a 21.02.0 build earlier and got a different error,
curl: (77) CA signer not available for verification

I've not made any changes on the server other than the certs test I was asked to do above. Things just stopped working and it's been like this since.

1 Like

The curl on your device is using PolarSSL rather than OpenSSL as we've been assuming. I'm guessing PolarSSL has similar issues to OpenSSL, but I'm not sure the proposed workaround will work. I don't know anything in particular about PolarSSL.

The output of the grep command suggests your device doesn't have the ISRG Root X1 certificate at all, which would mean there is no easy fix other than going into the field to update the devices. However, you mentioned this is an old device. Can you run the same grep command on a more up-to-date device?

2 Likes

Yes, I'm trying to get a hold of a newer one now and will run the same tests once I have one running.

1 Like

While you're work on that, can you also post your actual domain name? We'll need that in order to verify the actual certificate chain being served.

1 Like

I cannot post that, it's not mine to share. I can get what ever info you need using something like SSL Checker if you want? Which part do you need?

1 Like

download.openwrt.org (openwrt source/package repo uses LE too, but they chooes to use short chain. maybe it need short chain?

1 Like
Chain Information
Certificate # 1 - Common Name: domain.com
Decode this certificate for verbose information →
Subject Common Name 	domain.com
Issuer Common Name 	R3
Issuer Organization 	Let's Encrypt
Not Before: 	Sep 30, 2021 16:12:06 GMT
Not After: 	Dec 29, 2021 16:12:05 GMT
Signature Algorithm: 	sha256WithRSAEncryption
Serial Number: 	xxx
SHA1 Fingerprint: 	27:A6:46
MD5 Fingerprint: 	C4:EF:39
Certificate # 2 - Common Name: R3
Decode this certificate for verbose information →
In place? 	Yes, this certificate directly certifies the preceding one
Subject Common Name 	R3
Subject Organization 	Let's Encrypt
Issuer Common Name 	ISRG Root X1
Issuer Organization 	Internet Security Research Group
Not Before: 	Sep 04, 2020 00:00:00 GMT
Not After: 	Sep 15, 2025 16:00:00 GMT
Signature Algorithm: 	sha256WithRSAEncryption
Serial Number: 	xxx
SHA1 Fingerprint: 	48:78:2C
MD5 Fingerprint: 	5D:7C:43
OpenSSL Handshake

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain.com
verify return:1
CONNECTED(00000003)
OCSP response: no response sent
---
Certificate chain
 0 s:/CN=domain.com
   i:/C=US/O=Let's Encrypt/CN=R3
-----BEGIN CERTIFICATE-----
MIIFLjCCBBag
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gA
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=domain.com
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3316 bytes and written 302 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: xxx
    Session-ID-ctx: 
    Master-Key: 6FF36...
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000...

    Start Time: 1633047903
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
DONE

Does this help at all?

1 Like

Can you post what is in that /etc/ssl/certs/ca-certificates.crt file on the device(s)? Or at least its file size, or the number of times "BEGIN CERTIFICATE" appears in it? What I'm afraid of is that it may only have the expired cert in it, in which case I don't know what you'd do. But if it has a lot of roots in it, even if ISRG Root X1 isn't in there then you may have some options for other CAs.

1 Like

Ok, here is the newer version, the majority of devices that we have running.

% curl --version
curl 7.60.0 (mipsel-openwrt-linux-gnu) libcurl/7.60.0 mbedTLS/2.16.8
Release-Date: 2018-05-16
Protocols: file ftp ftps http https
Features: IPv6 Largefile SSL

% grep cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 /etc/ssl/certs/ca-certificates.crt

No results.

% ls -la /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 258424 Jul 8 2019 /etc/ssl/certs/ca-certificates.crt

certs.txt (256.3 KB)

2 Likes

The confusing part to me is all our sites are using letsencrypt certs and all of those are broken when using curl on openwrt. However, others like google, facebook, pinterest, none of those are broken. I also tested every same site using curl on Linux and Windows and those are all working. Only openwrt curl is not working, or at least, all I know about right now.

1 Like