Windows: invalid cert when fullchain.pem ISRG Root X1 is cross-signed by expired DST Root CA X3?

Help
1

My domain is: crt.sh | 6401694660

My web server is: Apache 2.4.51

The operating system my web server runs on is: openSUSE 15.3

I can login to a root shell on my machine: Yes

I'm using a control panel to manage my site: No

The version of my client is: python2-certbot-1.4.0

This certificate has been running fine since installation, as have all previous renewals. In our Apache config, we set SSLCertificateFile to the fullchain.pem file that is provided by Let's Encrypt. That file contains these three certificates in this order (I'm showing decoded snippets, the file contains PEM obviously):

#1 Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:95:c4:28:bb:82:76:f6:16:2e:ae:f7:0c:1e:4b:a3:e1:39
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Mar 24 01:03:18 2022 GMT
            Not After : Jun 22 01:03:17 2022 GMT
        Subject: CN = test.cookc.patientexp.com
#2 Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
#3 Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

Note: the last certificate in the chain is the version of ISRG Root X1 that is cross-signed by the expired DST Root CA X3. I know that shouldn't matter, but see below...

There is another Windows machine on the network that communicates with this machine via HTTPS, and that has been working fine until that Windows machine was updated a few days ago (our Apache server was not changed at all).

After the update, the Windows machine started failing HTTPS connections due to error validating the certificate.

In the heat of the moment, we fixed this problem by installing the ISRG Root X1 intermediate certificate into the Windows "Intermediate Certification" folder. After doing that, things immediately started working so the problem was resolved - but not fully understood.

Afterwards, we verified that the Windows machine does have the ISRG Root X1 certificate installed in its "Trusted Root Certifications" folder, but it has the self-signed version, whereas the version we are providing in fullchain.pem (which comes directly from certbot) is the version cross-signed by the (expired) DST Root CA X3.

So the immediate problem is resolved, but the lingering question is: why did Windows fail to validate the certificate in the first place when ISRG Root X1 was in fullchain.pem and in the Windows certificate store?

Does it have something to do with the fact that the two certificates were different variants, one self-signed, while the other was cross-signed by an expired cert? Obviously that shouldn't matter...

Has anyone else run into a similar sounding situation?

Thanks for any insights.

2

This may help...

3 Likes
3

Thanks. In this case, the Windows machine has the updated root already. That's what makes the failure mysterious.

2 Likes
4

Which version of Windows?

1 Like
5

When a windows machine doesn't trust a perfectly good certificate the answer is usually either:

  • the machine is not receiving windows updates or it's has firewall blocks preventing basic certificate updates.
  • group policy is set to prevent CA certificate updates (a common problem caused by a bug years ago which led to many administrators to switch the feature off, then not re-enabling it). See Turn off Automatic Root Certificates Update in group policy.

See also: An automatic updater of untrusted certificates is available for Windows Vista, Windows Server 2008, Windows 7, and Windows Server 2008 R2

The problem is possibly the R3 intermediate. You should have the R3 "Issued By" ISRG Root X1 under Intermediate Certification Authorities, which then chains up to ISRG Root X1 (Issued by ISRG Root X1).

The confusing part is there is also ISRG Root X1 (Issued by DST Root X3) and the corresponding R3 issued by that, but Windows doesn't trust that path due to the expired root, so it needs to be able to build Leaf > R3 > ISRG Root X1 (and it normally does exactly that as long as it has the correct intermediate available in it's store).

3 Likes
6

The problem is possibly the R3 intermediate. You should have the R3 "Issued By" ISRG Root X1 under Intermediate Certification Authorities, which then chains up to ISRG Root X1 (Issued by ISRG Root X1).

Yes, it looks like we are sending the correct R3 - i.e., the one signed by ISRG Root X1.

The confusing part is there is also ISRG Root X1 (Issued by DST Root X3) and the corresponding R3 issued by that, but Windows doesn't trust that path due to the expired root...

Well this is interesting. You are saying that Windows doesn't trust an intermediate certificate if it's signed by an expired root. But that makes me wonder if (due to the same logic) Windows might not trust a root certificate if it's signed by an expired root (even if a self-signed version of that same root exited in its store).

But I guess I will just need to do some tests somehow. Unfortunately this is a customer's production machine so we can't really play with it.

Which version of Windows?

Actually not sure yet, trying to get that information.

1 Like
7

There is only one R3 certificate issued by ISRG Root X1, since the public key is the same in both ISRG Root X1 certificates (self signed and cross signed)

There's also an R3 certificate issued directly from DST Root X3, and that might be a problem -- even though that certificate has been retired a while ago.

2 Likes
8

There is only one R3 certificate issued by ISRG Root X1, since the public key is the same in both ISRG Root X1 certificates (self signed and cross signed)

Correct - but one hypothesis (for which I have only circumstantial evidence of so far) is that Windows may have a bug where when it checks whether a certificate is in its store, it's checking for an exact byte-for-byte match of the whole file, instead of checking just that the certificate proper is the same and ignoring who signed it.

That would explain how when fullchain.pem contained ISRG Root X1 signed by DST Root CA X3, but the Windows CA store contained ISRG Root X1 self-signed version, validation incorrectly failed. Again, that's just a hypothesis at this point...

9

The public key in both R3 certificates is the same. But the signature is different.

The ISRG Root X1 signature will validate, the DST Root X3 won't.

But R3-signed-by-DST hasn't been around in a while. I don't know if that's the issue.

1 Like
10

@9peppe Thanks yep, you're absolutely correct, I'm getting it mixed up with the old one (which from memory was actually newer than the new one, lol).

@archie172 regarding which chain you are serving via apache - windows doesn't care it actually builds it's own `path from Leaf to Root using it's own store, so it only cares about finding a valid match for the issuer of the Leaf/end-entity cert which then leads to a valid root. There are no likely bugs in Windows certificate path validation (that I know of), but there are differences in behavior compared to things like OpenSSL. So, if the windows machine has the current R3 (under Intermediate Certification Authorities) and the ISRG Root X1 (under Trusted Root Certification Authorities) then it should normally work. Note that windows has user stores vs machine store so you need these to be in the machine store ("Local Computer"), not just the user store.

Note also that the assumption here is that the Windows machine is connecting to your apache https service using conventional windows APIs, not using something that bypasses those (e.g. Firefox, anything that embeds OpenSSL etc, as otherwise these would all be maintaining their own trust stores separately).

5 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.