CA certificates now redirecting to HTTPS

[encrypter1]

This isn't a support request, so much as filing a bug with Let's Encrypt itself, but since issues aren't accepted by email, I'm posting here and hopefully an engineer will see.

It seems that as of sometime recently, http://letsencrypt.org/certs/trustid-x3-root.pem.txt now redirects to HTTPS automatically and cannot be accessed over HTTP.

This is obviously stupid, since the whole point of this URL is to allow clients to obtain CA certificates they can trust, which MUST be done over HTTP.

We could obviously self host the certificate over an HTTP endpoint, but this has probably broken a lot of people's set up so this should be fixed globally. Hoping this can be done soon as we are seeing a lot of provisioning failures due to this issue.

[mcpherrinm]

Can you say more about your use case? I would not expect anyone to have an online dependency on our website.

Roots are generally distributed as part of your operating system or device firmware. Updates should be handled through a secure channel and verified manually, not blindly pulled over the internet via an untrusted plaintext connection.

[encrypter1]

The use case is for loading trusted CAs into embedded devices that do not trust Let's Encrypt as a certificate by default. This is somewhat common, particularly with older systems that don't know about Let's Encrypt from the factory. There are no "updates" for getting new root certs, this is the mechanism they provide for loading additional certs, through an HTTP URL.

These devices won't pull certificates over HTTPS since there's a chicken and egg problem.

We've been doing it this way for years, as have lots of other providers, and only some point recently does this seem to have stopped working, which we learned about due to end users letting us know that certain devices were not provisioning with TLS enabled. Right now, they're disabling encryption since the devices are rejecting Let's Encrypt certificates, due to this root cert not loading.

This isn't just me, this is a common practice in the industry as far as I'm aware, to support devices that don't know about Let's Encrypt in the industry. We could host it ourselves, but the risks of doing so are likely the same regardless of whether it's pulling from Let's Encrypt's server or ours.

[Nummer378]

@encrypter1

There are no "updates" for getting new root certs, this is the mechanism they provide for loading additional certs, through an HTTP URL.

This is incredibly insecure and only slightly better than just not using TLS in the first place. Any party can just insert their own root certificate into that HTTP stream, which then gets automatically trusted. You could just as well disable TLS authentication altogether.

@encrypter1

These devices won't pull certificates over HTTPS since there's a chicken and egg problem.

If you really need to download roots via HTTP(S), the best way to go would be to use a HTTPS server that uses a certificate signed by a root that's pre-installed on the device. The device should have been bootstrapped with something useful (ideally, a company's internal root certificate) that could be used for this purpose.

@encrypter1

t seems that as of sometime recently, http://letsencrypt.org/certs/trustid-x3-root.pem.txt now redirects to HTTPS automatically and cannot be accessed over HTTP.

It's also worrying that you seem to be looking for DST Root CA X3 which expired in 2021 and is soon going to be fully decomissioned. Any device should have loaded ISRG Root X1 years ago and no validation should be done against DST Root CA X3.

@encrypter1

This is somewhat common, particularly with older systems that don't know about Let's Encrypt from the factory.

If you're really loading DST Root CA X3 on these devices: That root certificate is approximately 23 years old, so "knowing Let's Encrypt" isn't part of the equation here.

---

I don't mean to impolite, I understand that you're just looking for a simple solution to a problem. I'm just shocked about the security state presented here: I was aware that embedded devices had bad security, but it's really that horrible?

[encrypter1]

This is incredibly insecure and only slightly better than just not using TLS in the first place. Any party can just insert their own root certificate into that HTTP stream, which then gets automatically trusted. You could just as well disable TLS authentication altogether.

Unfortunately, that's the way that it is. Fortunately, most of these are slightly older models, I don't see this as much anymore with some of the newer ones.

If you really need to download roots via HTTP(S), the best way to go would be to use a HTTPS server that uses a certificate signed by a root that's pre-installed on the device. The device should have been bootstrapped with something useful (ideally, a company's internal root certificate) that could be used for this purpose.

Only commercial certificates would likely be trusted for that, so that would be an additional expense. An internal cert wouldn't be trusted either because again, it would need to be loaded from somewhere.

It's also worrying that you seem to be looking for DST Root CA X3 which expired in 2021 and is soon going to be fully decomissioned. Any device should have loaded ISRG Root X1 years ago and no validation should be done against DST Root CA X3.

Well, that's a fair point, we've definitely had this unchanged for a while.

I don't see the actual certificate linked in either post though. I took a look here: Chain of Trust - Let's Encrypt

However, I'm having difficulty finding a link that matches the same format as above, i.e. with just the certificate, do you know which one would be the right replacement?

I don't mean to impolite, I understand that you're just looking for a simple solution to a problem. I'm just shocked about the security state presented here: I was aware that embedded devices had bad security, but it's really that horrible?

Unfortunately, yes, particularly for older ones. Ideally, it wouldn't be like this, but we have to support a wide variety of equipment and some models out there do have this stipulation. I'm definitely not a fan of it either, it would be nice if it "just worked".

[petercooperjr]

I think the link you're looking for is http://x1.i.lencr.org/ (which I got from the AIA CA Issuers from the R3 intermediate). That downloads the Root X1 over http. But everything said above about this being weird and not actually secure to use for blindly loading into a trust store certainly applies.

[encrypter1]

Thanks, but that appears to be a binary file, not a text file like the original link?

[petercooperjr]

Yeah, it's probably straight DER encoded instead of PEM-encoded. I don't know if there is a http link for a PEM-encoded one. You could test to see if the device accepts a DER-encoded file, though.

It does feel like whatever mechanism you would be using to try to put this URL onto devices, you could instead just put the whole root onto the devices instead. But that might be too logical for how these embedded devices end up working in practice.

[rg305]

@encrypter1

Only commercial certificates would likely be trusted for that, so that would be an additional expense.

You only need to find a secure distribution point.
That means any web server [on the Internet] that you control and is currently using a "trusted" certificate.
I have no idea which certs your devices trust, but AWS, Google, Cloudflare should be on that list.

[mcpherrinm]

The right thing to do here is to include the root in your next firmware upgrade.

Otherwise you should self-host it, as this isn’t something we are going to guarantee is going to keep working.

[encrypter1]

Unfortunately, we don't control the firmware, this is all proprietary software and this is the only mechanism that allows for using certs it didn't ship from the factory with. The device in question is out of support so there aren't any new firmware releases anyways.

I'll give the DER file a shot though and see if it'll accept it, if not, where might I find the PEM equivalent of it? If I have to self-host it, I guess that's what we'll do, it sounds like the X3 cert is an older one though so we should swap it out anyways.

[petercooperjr]

@encrypter1

where might I find the PEM equivalent of it?

The Chain of Trust Root Certificates page, you're looking for the ISRG Root X1 "Self-Signed" line, which has links to DER, PEM, and an openssl-text-style dump.

In general though, making a good trust store is a sophisticated thing. You need to make sure that you have a good plan for what you'll be doing if you need to switch to another CA, like if Let's Encrypt has an incident requiring it to make a new root certificate (or plain just runs out of funding).

[mcpherrinm]

Note that "PEM" is just base64-encoded DER (binary), along with a header and footer text lines, so it is easy to convert from one format to the other, if needed. There are many tools online to do so.