Renewed certificate does not work. Please help

Help
1

Hi guys,

I'm not an expert in this matter but I'm currently the only one leading the project and I desperately need help to make this work before June 24th.

We run an MQTT broker on port 8883 on AWS Elastic Beanstalk (Node.js runtime) (details below).
The certificate is expiring on June 24th so I renewed it following the steps below also.
The problem is that the IoT devices are not establishing a successful TLS connection with this new certificate and the only error message on my end is the following:

{
        "type": "Error",
        "message": "140140484839296:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:308:\n",
        "stack": "Error: 140140484839296:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:308:\n",
        "library": "SSL routines",
        "function": "ssl3_get_record",
        "reason": "wrong version number",
        "code": "ERR_SSL_WRONG_VERSION_NUMBER"
    }

The root certificate included in the devices' firmware expires in September. I assume it's also issued by LetsEncrypt, not sure what details do you need to validate this. This is the confusing part to me.

Questions:

  • How can I check that the existing root certificate accepts the renewed certificate in the broker?
  • Is the renewed certificate invalid or is this just an issue with the TLS version? Not sure what to do in that case.
  • Should I renew the root certificate now? (I should before September and I don't even know how to do it and I want it to last at least 5 years)

I'd appreciate your help guys. Thank you very much.

My domain is: broker.aritronix.com

I ran this command:

  • sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
  • sudo yum-config-manager --enable epel
  • sudo yum install certbot
  • sudo certbot -d broker.aritronix.com --manual --preferred-challenges dns certonly

It produced this output: Do not remember but the certs were issued with no errors or warnings.

My web server is (include version): nginx, but not in use since we run an MQTT server on port 8881/8883

The operating system my web server runs on is (include version): Amaxon Linux 2

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): AWS console

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): -bash: certbot: command not found. Not sure why it is not there anymore, maybe the instance was recreated after the certificate was issued.

2 Likes
2

The certificate currently installed on ports 8881 and 8883 was issued on March 26.

I can see that you have issued newer certificates for that domain, but the MQTT server is not using them. You probably need to reload/restart it in order to pick up the new certificate.

How recently did you start seeing that error?

What version of Node is this error message coming from? Can you post the part of the code that establishes the connection?

I ask because the certificate (from March) looks OK and I can connect to your broker on both ports just fine.

There's a minor problem that you're missing the intermediate certificate on port 8881, but that shouldn't cause the version error you posted.

If you're explicitly trusting root certificates on your IoT devices, you should probably add "ISRG Root X1" and "ISRG Root X2" from Chain of Trust - Let's Encrypt.

2 Likes
3

Yes, the current broker is using the old certificates because we had to deploy and rollback after we saw that devices were not connecting.
So that I recreated a new environment in staging.broker.aritronix.com with specific certificates for that domain. A test device is trying to connect to that broker, BUT it is using the same root certificate for the original domain, so that can be the reason it doesn't work. SHIT.
Can you please check the new certificates for staging.broker.aritronix.com?
What info do you need from the root cert to make sure it should work with the renewed certificate?
Thanks.

EDIT: using Node.js v12 and aedes MQTT broker

2 Likes
4

Thanks!

About root certificates, if you take a look at OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates, you'll notice that "OpenSSL (any version) without ISRG Root X1 in trust store" is not compatible with certificates renewed after June. This somewhat depends on what your client code is doing (whether you're passing a ca option to the tls options).

Thing is though, if you were having trouble with that, you'd be getting errors from Node.js about verifying the certificate. That's not what we see, so I don't think trust is the exact issue.

I tried also running this program from Node.js v12, and it gets past the SSL handshake just fine. You are using Node v12 on your IoT devices, right? Not just the server?

const mqtt = require('mqtt')

const client = mqtt.connect({
        host: 'staging.broker.aritronix.com',
        port: 8883,
        protocol: 'mqtts'
})

client.on('error', (e) => {
        console.log(e)
})

client.on('connect', () => {
        console.log('Connected!')
})

Out of curiosity, how much memory do these IoT devices have?

There was one other thread here where the person had a similar kind of issue, and it turned out that the extra certificate in the post-June certificate chain, caused their device to run out of memory and crash when trying to verify the certificate chain. They solved the issue by not sending the full size chain. However, they were using an actual microcontroller. Since you're running Node.js, I'm guessing you have more memory available than just mere kilobytes.

2 Likes
5

I realize I may have made some wrong assumptions about your setup.

I interpreted your original post to say that you were getting that SSL protocol error on the IoT devices themselves.

Now, I realize you may have meant that you are seeing that error on the server side when the devices connect, but you don't know what's happening on the IoT devices themselves.

In that case, the running-out-of-memory theory might be likely, if you're running on 128KB RAM devices or whatever.

2 Likes
6

What microcontroller are you using that is falling over?

2 Likes
7

Thanks for your help guys, but there are so many terms I don't understand here.
The root certificate that the devices appear to have is:

Common Name: R3
Organization: Let's Encrypt
Country: US
Valid From: October 7, 2020
Valid To: September 29, 2021
Issuer: DST Root CA X3, Digital Signature Trust Co.

Does that tell you anything useful?

There are a lot of different hardware combinations out there, it is hard to say how much memory they have.
Additonaly, those devices used to have 3 root certificates before the migration to the custom broker solution, so hard to think it is a memory issue.

I'm still stuck and I need to figure out which certificate is wrong. The firmware upgrade for thousands of devices are manually done and it could take like 2 months. We should start ASAP if this is the case.

Can someone please recommend which root cert to use after all this shit is magically fixed and before September?

Thanks.

1 Like
8

R3 is an intermediate, not a root, and the R3-signed-by-DST-Root-X3 certificate is no longer being sent in the default chains from the ACME Servers. If your IoT devices are expecting that specific certificate, then that's why they're all broken. That would be a horrible misconfiguration.

The root that should be in their trust store is the self-signed ISRG Root X1 (and if you want a bit of future compatibility, the self-signed ISRG Root X2 as well), available from the top of the Chain of Trust page. Also, you probably want at least one "backup" CA in there too, possibly one you run yourself, just in case Let's Encrypt has some (temporary or permanent) problem issuing certificates, due to compromise, distrust, or something more mundane like running out of funding.

Then when a client connects, your server needs to send the full "chain" of intermediates up to the trusted root. Intermediates can change at any time.

1 Like
9

This is an intermediate certificate (the old R3 cross sign), one that is being phased out really soon - you will already get a different one on renewals - this may be related to the issues you're having.

The IoT devices must not pin any intermediate certificate. The actual root is ISRG Root X1. For an overview on how the hierarchy looks, have a look at the Chain of Trust page.

PS:

If your IoT devices validate expiry dates on certificates, the intermediate you are currently pinning will expire on Sep 29 19:21:40 2021 GMT - after this date there is no chance to employ any certificate workaround to still use the old R3 cross sign.

[Your leaf certificate will expire sooner, but there are workarounds to serve new renewed leafs with the old chain]

1 Like
10

Thanks guys, but still, I'm not sure how to proceed. I need a way for those devices to connect until the end of August at least so I can plan the upgrade to the new certificate.
What can I do server-side to allow the current devices to connect? There is no time to upgrade before June 24th. Should I generate other certificates?

I'd appreciate your help one more time.

1 Like
11

Which one of these certs should I use?

Self-signed or cross-signed?

1 Like
12

You want the self-signed version, and then when running your ACME client on your web server you want to select the "alternate" chain. (The "main" chain will be sending DST Root CA X3 even after it expires in a creative workaround to try to keep old Android versions working on typical user-facing web servers, but for most "programmatic"-type use you want the "alternate" chain which just uses a normal root rather than an expired one.)

1 Like
13

You may be able to update your server so that it serves your leaf along with the R3 cross-signed-by-IdenTrust intermediate (also available at that Chain of Trust page).

No, that won't help you at all.

1 Like
14

You may be able to update your server so that it serves your leaf along with the R3 cross-signed-by-IdenTrust intermediate (also available at that Chain of Trust page).

I'm trying hard to understand what you mean. certbot generates 4 pem files, which of those should I use and which one should I replace with the one you mentioned above to make this work?

Thanks again.

1 Like
15

Assuming your web server references fullchain.pem already,

  1. Rename the fullchain.pem that certbot gave you (just to have as a backup)
  2. Concatenate the contents of cert.pem (which is your leaf certificate) and the contents of https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem together into one file, which will be your new fullchain.pem file. So it should have two certificates total in it, each of which starts with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----.
  3. Restart your web server to pick up that fullchain.pem file.

Again, this is a temporary hack that would only work for the next couple months, and is making a lot of assumptions about how your clients are relying on sending that exact intermediate along. Based on what you've told us so far I'm not sure it's actually what your problem is. You just said that your clients were trusting R3, directly, which set off all sorts of alarm bells.

1 Like
16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.