Should I add both DST Root CA X3 and ISRG Root X1 intermediates?

Hi,

So I have multiple domains running with Lets Encrypt certificates without any issues. But now we have a new domain where we see some strange behavior. We have an agent that is deployed across computers to report in data to our server which uses the https route. And within one organization some of the computer succeed to validate the cert and some fail.

On the working machine. Browsing to https://valid-isrgrootx1.letsencrypt.org/ and checking the certificate path chain it shows DST Root CA X3 followed by Let’s Encrypt Authority X3 then valid-isrgrootx1.letsencrypt.org and all certificates are reporting as OK

From an none working machine. Browsing to https://valid-isrgrootx1.letsencrypt.org/ and checking the certificate path chain it shows ISRG Root X1 followed by Let’s Encrypt Authority X3 then valid-isrgrootx1.letsencrypt.org and all certificates are reporting as OK

If I check our fullchain.pem/chain.pem those only contains the DST Root CA X3 intermediate, and my guess is that is why it is failing?

Our certs are generated by running the certbot and manually configured Apache which provides each of the certificates including chain/cert/key and running on Ubuntu 14.04.

So I’m trying to understand a few things here:

  1. Why are some computers trying to validate against X1 instead of X3, and is that something we can control from the server?
  2. Can/should I add both intermediates to the chain files? If so, how do I do that using certbot?
  3. If not 1 or 2 will solve it, what should I do instead?

I have tried hard search the internet and your forum for this and not found any similar case.

Thanks!

Hi @gsus

what's the domain name?

The DST Root CA X3 is a root certificate, not an intermediate. So it should be preinstalled.

But your server should send the intermediate certificate.

Perhaps check your server with SSL Server Test (Powered by Qualys SSL Labs)

There you should see a path like:

  1. Sent by server
  2. Sent by server
  3. In trust store

Not good is 2. Extra download.

Hi @JuergenAuer,

Thanks for your reply!

I have checked the ssllabs and it reports everything is fine.

Yes, but as I have understood it, each root cert is connected to an intermediate. So if I am to enable 2 root certs for validating my cert then I would need both of those intermediates. But maybe I have misunderstood how it works?
Our server do send 1 intermediate, but I wonder if it would need to send an intermediate for each root cert, that is sending 2 different intermediates instead of 1. Because if I look at ssllabs I see only the X3 as root in the patch for Windows:

But if I check https://www.ssllabs.com/ssltest/analyze.html?d=valid-isrgrootx1.letsencrypt.org
I see both of them:

So to sum it up:

  1. Why do computers chose a root cert that is not in my path?
  2. How do I solve this so that they either always chose the X3 OR our server will also approve X1 validation?

Thanks for your help! Really appreciate it!

Hi,

First of all, ISRG Root X1 is not trusted by Microsoft Windows now.

If a server send LE intermediate signed by ISRG Root X1,
browsers on Windows downloads LE intermediate signed by DST Root CA X3 showed in end-entity certificates.
But I think this method is available on only Microsoft Windows.
For example, Android5 shows error on https://valid-isrgrootx1.letsencrypt.org/

The only reason why the second path of the SSLLabs test actually works is because the end certificate for https://valid-isrgrootx1.letsencrypt.org has http://cert.int-x3.letsencrypt.org/ as value for the CA Issuers label in the Authority Information Access extension, which is the X3 intermediate signed by DST Root X1.

That way, if the intermediate certificate actually sent by the server (in this case signed by the ISRG Root X1) doesn’t resolve in a trusted path, a client could try to fetch the other intermediate certificate as reported in the end certificate like I explained above.

If Let’s Encrypt really wanted to let clients test the ISRG Root X1 certificate, it should have removed or fixed the reference to the DST Root X1-signed intermediate.

Thanks for your reply guys! I really appreciate it!

It still boils down to:

Why does computer try the X1 when I send the intermediate for X3?
And the most important question: How do I fix it and can it be fixed on the server side at all?

Thanks!

I didn't find the actual difference between the working and non-working clients in the thread. Are they a different OS? Or perhaps different versions of the same OS?

Perhaps the non-working clients have a problem with the DST Root X3, but we can't know for sure if we don't know without more information about the clients

That I don’t know since I don’t control the machines. But we have thousands of computers reporting in through different domains, all using the same certbot config to generate the SSL, and I have never heard about this issue until this specific customer. That doesn’t mean it doesn’t exist on other customers, but I get a feeling I should have come across it if it was common at least. So my initial feeling is that there is something on the client side that makes it go for the X1 certs instead of the X3 certs. I’m working with their technicians to try and find the cause, but I can only search on my side so to speak, that being the server and its certificates. They are all Windows machines in the same AD, but they do have a bunch of security suits. For example Barracuda Web Security, which seems to have some functionality to validate SSL so we are looking into if that is messing with this as well. They also have BitDefender and McAfee as well as standard Windwos Defender, which I guess could also lead to some unexpected behavior when mixing so many security suits.

They tried to add both the X1 and X3 to their trusted root store already, and that did not help.

Both showing OK when they go to https://valid-isrgrootx1.letsencrypt.org/ and inspecting the certificates, but they get the different ones. Failing computers get the X1 and succeeding computers get the X3.

The error from cpprestsdk is this: WINHTTP_CALLBACK_STATUS_FLAG_CERT_REV_FAILED failed to check revocation status.

Is there anyway I can pass something to certbot to make it include both intermediates to solve this on the serve? If it is possible to support both of them, that would of course be the best approach and also making it future proof I guess. But maybe that will introduce different kind of problems?

I think certificate verify path depends on clients.

For example,
I think verifying path used by Mozilla Firefox depends on the browser user activity.
But I know about this only a little.

I am using ISRG Root X1 chain to verify certificates sent with intermediate signed by DST Root CA X3 in Firefox Stable and Nightly.
I am using DST Root CA X3 chain to verify certificates sent with intermediate signed by DST Root CA X3 in Firefox Developer Edition.

No, that won't work. Each certificate in the chain must have signed the certificate immediately preceding it. A grab bag of intermediates isn't valid. Some clients will ignore the error, and some will fail to validate it.

Thanks for all replies!

According to this image they are crossed signed, and not in a straight chain as I understand it. Doesn’t that mean that they should be able to be validated with either the X1 or X3? I thought if one fails, it should try the other, and either should be able to validate the cert. Or am I misunderstanding it?

If so, then why does it fails to validate against the X1 if the X1 is trusted by both the server (not an intermediate) and the client (as seen when going to the validation site in earlier example).

That image is misleading: There is two different "Let's Encrypt Authority X1", one trusted by "ISRG Root 1" and another by "DST Root CA X3"

(but, because they share the same key and name, both are recognized as valid for the leaf certificate)

No, an intermediate can only be validated against one root. That's why there is one X1 intermediate valid for the X1 root, and another X1 intermediate valid for the X3.

You server should send to the server the one that it thinks will be recognized by the client as trusted.

The client may decide to use another path. For example, if your server sent "Let's Encrypt Authority X1 signed by ISRG Root 1" but, the client still have in cache (because another website sent it earlier) "Let's Encrypt Authority X1 trusted by DST Root CA X3", the client may use that one instead.

The roots trusted by the server doesn't matter.

It may fail if your client trust the X1 root but didn't received the intermediate that links to it.

Your client may also fetch the intermediate using the caIssuers field (1.3.6.1.5.5.7.48.2) of your certificate.

Did you investigate more that error? It means that the client can't check if the certificate have been revoked. To check it for a let's encrypt certificate, the client need to be able to fetch OCSP. If your client failed to fetch it, it may decide it's a fatal error. You can force your server to send OCSP information to avoid that using OCSP stapling.

Thank you so much for you detailed answer!

I will look more into the revocation error, but I think that is just a side effect from trying to validate against the wrong root cert.

Is there anyway to support both root certs or to force the client to skip cache and chose the X3 root cert for validation? I guess on solution could be to remove the ISRG from the trusted root store in the AD for that customer, right?

Yes, it should work. But I'll not recommend it:

You can't force the client to choose a path for validation. But, as long as you give him one valid path, it doesn't impact other part of the certificate validation (revocation, ...)

Indeed, the revocation of the certificate is done using the information provided by the certificate, not the intermediate used in the path.

The only thing (if I'm not mistaken) that the validated path impact is certificate pinning, if you used a method to pin root or intermediate (in that case, you could pin multiple roots/intermediates).

So after more research I found this:

It seems like when they got the ISRG root cert into Windows trusted store things started to break since computers will now start to try and validate against the ISRG root cert which is not part of the default chain which seems to break it. Not sure if this is an issue within cpprestsdk, Lets Encrypts certs or a configuration somewhere. But my guess is that I will not be the only one running in to these issues in the coming days. I will investigate this further, but I really appreciate any input people might have.

Is it possible to switch over and use the ISRG cert path instead of the DST? If so, how is that done?

Microsoft has to deploy that -> next patchday. My Windows 10 doesn't have it.

A Letsencrypt-certificate, created 2018-06-15 ( https://www.server-daten.de/ ). My FireFox shows the ISRG root, my Chrome (+ IE + Edge) shows the DTS root.

It's the same intermediate certificate with two different roots.

Try https://valid-isrgrootx1.letsencrypt.org/ with IE or Edge. Since Windows 7 root certificates were downloaded on demand. My Windows 10 has ISRG Root X1 already.

1 Like

I see (Edge) again DTS Root CA X3 as Root certificate (browser view).

The Certificate Store hasn't the certificate (not user, not machine).