IOT Devices with X3 certificate embedded

Help
#1

My domain is: gotailwind.com

We have LetsEncrypt certificates embedded in one version of firmware that has been working fine for years, and it is based on the "X3" (forgive me if I'm saying this incorrectly). When I run the OpenSSL command to check the certificate expiration
openssl x509 -enddate -noout -in .pem it gives the output notAfter=Mar 17 16:40:46 2021 GMT. This leads me to believe it should continue to work until March 17th as long as we do things correctly. This gives us enough time to make sure all devices get the new firmware.

I've spoken with numerous software developers about the changes coming to LetsEncrypt on January 11th and cannot get a solid consensus on what we need to do to keep the devices with this firmware version working past the changes happening on January 11th.

I'm really hoping I can find someone who knows with certainty what the changes are, and what we need to do to validate and test before January 11th so the transition goes off without a hitch. I am willing to hire someone on a short-term contract to guide us and make sure we do the right things here. I don't want January 11th to be a bad day.

My web server is: NGINX (not sure of the version - I'm not a software developer personally)

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): Yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Don't know

2 Likes
#2

To clarify: Is the device with the firmware acting as an HTTPS client or a server?

From your description, it sounds like the device is configured as an HTTPS client, and is configured with only one trusted root certificate: "Let's Encrypt Authority X3." Is that accurate?

3 Likes
#3

Hi Jacob. Thank you so much for responding. Yes I believe that is the case.

3 Likes
#4

Unfortunately, you're going to run into problems earlier than you think, for a different reason. "Let's Encrypt Authority X3" is not a long-lived root certificate, it's an intermediate certificate. We issue from intermediates, but they are trusted because they are in turn signed by root certificates. Intermediates change more often than root certificates.

Specifically, on December 1 we started issuing from our newer R3 intermediate. That intermediate is signed by our "ISRG Root X1" root, just like Let's Encrypt Authority X3. But because your devices do not trust "ISRG Root X1," they will not trust certificates issued by R3.

Do your devices talk to a single API server? What's the hostname for that server, and how long is left on the lifetime for that server's certificate? This will become a problem for you when you next rotate certificates on that server. Normally that happens 60 days after the last certificate was issued, but in a pinch, you can stretch it out all the way to the NotAfter date on the certificate.

2 Likes
#5

connect.gotailwind.com and my understanding is the current certificate on there expires January 2nd. I was told we can update it at least one more time, which would have bought us until the end of February.

1 Like
#6

Unfortunately, that means your devices won't be able to connect to your server after January 2nd.

Before Dec 1, that would have been the case. But as of now, any new certificate you request will be issued from R3, which your devices will not trust.

To whatever extent possible, you should make sure your customers connect their devices to the Internet for a firmware update before January 2nd. I know this is a tough situation to be in - sorry about that!

3 Likes
#7

Ok thanks. I was afraid of that.

3 Likes
#8

I'm still going to have them run a test with the R3 to be sure. I also saw there was a mention of "disaster recovery" certificates but it sounds like those aren't going to help.

2 Likes
#9

If you can share more details about the "disaster recovery" certificates and how their used in your devices, it's possible those are what you need.

One other question: Are you using a custom root store, or are you doing key pinning? If you're doing key pinning (the term "disaster recovery keys" is commonly used in that context), you may have a path forward.

1 Like
#10

Honestly Jacob this stuff is all way out of my wheel house, which is why I was really hoping I could hire someone from the Let's Encrypt team to spend maybe 30 minutes to an hour looking it over. I could share the full details that way - even the certificate that's embedded in the firmware. I know there is only 1 certificate in that version of the firmware. When I mentioned "disaster recovery" I mis-spoke. I was actually referring to the LetsEncrypt R4 and E2, but those obviously do not apply in this case.

I know we were able to issue a new certificate somehow on December 6th that is valid until January 2nd (25 days apparently).

2 Likes
#11

Part of the confusion here is that there are two certificates with "X3" in the name: There's "Let's Encrypt Authority X3" and "DST Root CA X3". They have quite different purposes, but the replacement of each of them is on a different schedule (and yes, "Let's Encrypt Authority X3" is already not being used for signing new certificates). It's not clear to me which one (if either) is in your device's trust store, and it doesn't sound like you're sure either. I think we'd need more information on what you're doing to be able to help you.

I don't think Let's Encrypt itself has consultants for hire, though I believe there are some people here who do the occasional freelancing.

3 Likes
#12

Thank you for your reply Peter. That's useful information. I am trying hard to reach the developer that updated the certificate on the server on December 6th. Since December 6th is after the December 1st date Jacob mentioned, it makes me think our firmware is looking for the DST Root CA X3 then, because I know he generated another intermediate on the 6th.

2 Likes
#13

Here's what I can put together from what you've told us and what I can see from looking at your domain and various public places:

  1. Your company has, since mid-2017, been using Let's Encrypt to issue certificates for your API server at connect.gotailwind.com
  2. Throughout that same time frame, the IoT devices you have created have been told to "trust" just a single certificate: "Let's Encrypt Authority X3", which was the intermediate which issued all of your certificates so far.
  3. On December 1st, we changed our systems so that all subsequent certificates are issued by a different intermediate: "R3".
  4. On December 4th (exactly two months after your previous certificate was issued, as is completely normal), your certificate was automatically renewed, and therefore was now issued by R3, which your devices don't trust.
  5. One of your developers realized that something was wrong, and put your previous certificate (the last one issued from Let's Encrypt Authority X3) back on your server so your devices would continue to work.
  6. Therefore, on January 2nd, when that certificate expires, your devices will break unless you're able to update them first.

The problem here is that your devices only trust an intermediate. Unfortunately, intermediates are subject to change at zero notice (for example, if one of our intermediates was compromised and we had to switch to issuing from one of our disaster recovery backups), so only trusting an intermediate is a bad idea.

The solution here is to ensure that your devices instead trust a root, which is much less likely to change on short notice. (Or better yet, multiple roots, for safety.) I suggest that you update the firmware on your devices so that they instead trust both "ISRG Root X1" and "DST Root CA X3" at a minimum.

You can learn more about our certificate hierarchy, the difference between intermediates and roots, and more, in our public documentation: https://letsencrypt.org/certificates/

6 Likes
#14

Thanks for that detailed reply Aaron. Much appreciated.

2 Likes
#15

You may also want to check out this somewhat-recent thread about building a trust store for embedded devices:

The short of it being that in addition to trusting Let's Encrypt's root(s), you probably want to be trusting at least one other CA, possibly one you create yourself, just to protect yourself in case Let's Encrypt becomes unavailable on short notice. While I have every expectation that Let's Encrypt will continue indefinitely, if they suffer a major compromise that requires new keys, or suddenly run out of funding (it is a non-profit relying on donations), or otherwise stop issuing certificates due to whatever temporary (or permanent?) problem, then you want to have another way for your devices to authenticate that they're securely connecting to your systems.

3 Likes
#16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.