Modern browsers build their own chains. There's lots of logic involved including caching, pre-validation of CA signatures, preloading of CA certificates and many more.
All of this has the effect that the chain displayed by the browser has almost zero relationship with the chain send by the server - the browser effectively does what it wants. In fact, many browser configurations do not even care if you send the intermediates at all*. To diagnose certificate chains, browsers are the wrong tool nowadays.
Older, or more simple TLS clients still rely on the server sending a correct and valid chain - which you're probably doing.
*Some browsers not doing CA preloading need to "learn" the intermediate first, by receiving them once and then remember them (caching). In theory this doesn't need a certificate chain send by server, but can also be done via AIA loading, though I don't have knowledge how this is currently implemented in diffferent browsers.
If you used the tool I gave you and it says that your webserver is not serving the full chain then the problem is that your webserver is not serving the full chain. Your webserver should be serving all of the following (in this exact order):
DST Root CA X3 expires at 10 pm on September 30, 2021. I want to confirm whether it will affect my website? Will the validity period of the root certificate be automatically extended?
I have conducted additional testing for Windows server specifically and observed the following:
Windows may continue to serve Leaf > R3 > DST Root CA X3 after expiry if the expiring R3 is present in the Intermediate Certificate Authorities store at the level of Local System. In my testing the server did not automatically switch over to the non-expiring R3 given 30 mins after expiry. A reboot can be performed to force the correct R3 to be served.
Removing the expiring R3 cert manually can be done to expedite the changeover:
Powershell method:
Launch powershell as local system: psexec.exe -i -s powershell.exe
Run cmd as Administrator and delete the registry entry for the R3 intermediate in the Local System CA store: reg delete HKEY_USERS\S-1-5-18\Software\Microsoft\SystemCertificates\CA\Certificates\48504E974C0DAC5B5CD476C8202274B24C8C7172 /f
[/Edit]
If the correct chain is still not being served, edit an https binding in IIS (delete an https binding then add it again), alternatively you may need to reboot the server to force the new chain to be served.
Additionally it was observed that Firefox builds it's own chain (as already noted elsewhere) Leaf > R3 > ISRG Root X1 when the old R3 in Leaf > R3 > DST Root CA X3 expires, regardless of the chain served by the server.
We have a simple SSL certificate that we get from Let's Encrypt using certbot.
So the full chain PEM we get contains: server certificate -> R3 -> ISRG Root X1 (issued by DST Root CA X3).
Alternatively, clients (browsers, JRE, etc) can use the alternative chain: R3 -> ISRG Root X1 (self-signed).
Now here is my question: is there any way to make openssl s_client use the alternate chain ?
I'm asking this because we are monitoring our servers using openssl s_client -showcerts -connect ${SERVER}:${PORT} -servername ${FQDN} to retrieve all certificates sent by the server.
Then we check that the end date is not near for each individual certificate retrieved + the root CA.
So basically our probe will always be red as Let's Encrypt decided to deliver the ISRG Root X1 certificate issued by DST Root CA X3 (instead of the self-signed one).
Your server sends the certificate chain and this is what tools like OpenSSL display. If you want to change that you need to change what your server sends - this is server-side, not client-side. Modern browsers tend to build their own chains and ignore the ones send by the server - OpenSSL doesn't do this.
You could also consider to just fix your way of checking expiry dates: Any correct implementation will stop verifying at the first trusted certificate (trust anchor). For example, if you look at your openssl output, you will see something like this [output of -showcerts omitted].
CONNECTED(00000184)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = community.letsencrypt.org
verify return:1
---
Certificate chain
0 s:CN = community.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Note the verify output displayed above the chain; see how it stops verifying at ISRG Root X1 and doesn't check DST Root CA X3? That's because ISRG Root X1 is already a root and there is no need to go further up. If your checking tool can do the same, you won't have this problem.
It has never been clear to me how a Web Browser retrieves intermediate certificates if the server does not send them (i.e. Sent by server). How does a Web Browser know where to retrieve these intermediate certificates? I am curious so I can do the same retrieval to those intermediate certificates so I can manually add them to my server's fully chain if necessary. And this DST Root CA X3 expiration and other cross signing is a scenario where I may run into such a situation.
The intermediate certificates are supposed to be sent by the webserver. Your ACME client should give you all the necessary intermediate certificates with your leaf certificate when you request a certificate.
Correct; but in the past I have had a missing intermediate certificate and just want to know how to retrieve the intermediate certificate properly (and get the right one ) So I can correct it if I run into again.
This is supposed to be the job of the ACME client...which is to say the process is defined by the ACME protocol (specifically section 7.4.2).
To download the issued certificate, the client simply sends a POST- as-GET request to the certificate URL.
The default format of the certificate is application/pem-certificate- chain (see Section 9).
The server MAY provide one or more link relation header fields [RFC8288] with relation "alternate". Each such field SHOULD express an alternative certificate chain starting with the same end-entity certificate. This can be used to express paths to various trust anchors. Clients can fetch these alternates and use their own heuristics to decide which is optimal.
The certificate and the intermediate(s) are both provided when the client downloads the finalized certificate. If an alternate chain is available, the client is notified via the response headers for the cert download and can then download the alternate(s) as well. Most of the popular clients now support some method of specifying which chain you want to use.
Obviously, the chain details are public and Let's Encrypt provides alternative means to get them as @griffin pointed out. But if you can work with your ACME client to use the ones provided by the ACME server, you won't have to worry about updating them manually later.
This is where I'm confused, the ISRG Root X1 certificate sent by the server (which has been delivered by Let's Encrypt) is not a ROOT certificate AFAIK as he is not self-signed (signed by DST Root CA X3).
$ openssl x509 -noout -in /tmp/sent -subject -hash -issuer -issuer_hash
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
4042bcee
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
2e5ac55d
Command used on the actual self-signed ISRG Root X1 certificate :
$ openssl x509 -noout -in /etc/ssl/certs/ISRG_Root_X1.pem -subject -hash -issuer -issuer_hash
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
4042bcee
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
4042bcee
So, according to my understanding, the self-signed certificate is a ROOT certificate, but how could I determine that the ISRG Root X1 certificate sent by the server can be considered as root too ?
From a openssl point of view there is no hint right ?
The only way would be to reproduce what the browsers does: creating the alternate chain inside the probe algorithm ?
You've misinterpreted the OpenSSL output I'm afraid. As @arulibao already said, the ISRG Root X1 certificate in the example you've quoted is NOT a root certificate, as it's indeed not self-signed. It's signed by DST Root CA X3.
That said, there is a certificate with the ISRG Root X1 private key which is self-signed. Same Common Name, same private key (corresponding with the Common Name), but a different certificate altogether.
If the issuer and subject is the same, it's an indication it's a self-signed certificate. Strictly speaking you'd need to verify the signature of the certificate with the public key of the same certificate.
However, root certificates shouldn't be visible in the chain send by the server. So it's probably better to turn it around: if the issuer and subject are NOT the same, you're NOT dealing with a root certificate.
This isn't entirely correct and there seem to be some misconceptions here about what a root certificate actually is:
A root certificate is something that is in your trust store. It is not something send by the server.
You're both focusing on the fact that the server sends an apparent intermediate certificate called ISRG Root X1. But ISRG Root X1 is a known root and therefore it really doesn't matter what was send by that server - it's irrelevant. A server sending some certificate doesn't change the definition of what a root certificate is. ISRG Root X1 is a root certificate and this fact doesn't change by someone sending a version of ISRG Root X1 that appears like an intermediate. The purpose of this certificate is to aid clients that are not aware of the fact that ISRG Root X1 is a root and thus show those clients an alternate path to a different root.
The hint is the trust store. OpenSSL walks up the certificate chain by first looking at the leaf cert, seeing the issuer R3, which it doesn't know, but learns it because it was send by server. Then the issuer of R3 is looked up, which is in the trust store hence the trust anchor is reached and verification is stopped*. ISRG Root X1 signed by DST Root CA X3 is never used in the verification by OpenSSL versions with X509_V_FLAG_TRUSTED_FIRST **.
I have not. The verify output above the chain displays the verification path as OpenSSL walked the tree. DST Root CA X3 (and it's "intermediate, ISRG Root X1 signed by DST Root CA X3" is not part of the verification and this is shown (or not shown, as it's missing from the verify path).
You can compare this with older versions which do not support X509_V_FLAG_TRUSTED_FIRST and sadly also misunderstand the concepts of roots:
# openssl version
OpenSSL 1.0.1j 15 Oct 2014
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = community.letsencrypt.org
verify return:1
---
Certificate chain
0 s:/CN=community.letsencrypt.org
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
This OpenSSL version verifies up to DST Root CA X3, even with ISRG Root X1 in the trust store and this is shown in the above verify output. Newer versions always stop at ISRG Root X1, which can be seen from the verify output - it's unrelated to what chain you send.
**Assuming the trust store is up to date.
*This is simplified for better understanding and is in reality slightly more complicated. OpenSSL in fact first reads the chain as send by the server and then replaces certificates with the ones found in the trust store. Then signatures are verified.
Web browsers use a variety of strategies. One option is called AIA Chasing. The idea here is, a certificate, such as the leaf certificate for your server, has a section named Authority Information Access which indicates a URL where there's a certificate for the issuing CA. This certificate of course might in turn contain AIA. This has a small privacy impact, because during AIA chasing the CA learns that you enquired about some certificate it issued, and in theory they could set AIA to indicate separately each such certificate issued (Let's Encrypt of course does not) so that they could track exactly which sites are visited.
Another option is to remember intermediate CAs the browser encountered recently and use those to try to "fill in the gaps" when visiting a site that doesn't provide a chain. Because certificates are signed documents, it isn't important where you saw them, you know they're authentic, but this can mean sites work OK on a "warm" browser but don't work if visited from a new session or e.g. after rebooting.
Current Firefox versions provide a complete set of all unconstrained intermediates (e.g. R3) baked in the browser itself, so it will work with any normal public certificate even if your server didn't send the chain correctly.
Of course you should not depend on this in your site as doing so can only make it less reliable, and software other than web browsers rarely implements these strategies.
Hm, my bad, I assumed (incorrectly) that the root certificate used as trust anchor didn't show up in the verification list of certificates of the OpenSSL output. So I assumed that when it showed and stopped at the ISRG Root X1, it showed it as an intermediate certificate.
By the way, I'm not entirely sure if the definition of a root you're using here is also globally used. I've heard about trust stores assigning trust to just the private key instead of a (self signed) certificate, but if all trust stores work that way? I dunno..
I would argue that it's technically the public key, as you can't trust something you don't know. But yeah I see what you mean - definitions can vary.
A fact is that you need to name your keypairs - which is what the certificates commonly do (by having a Subject field). You can also use things like the Authority Key Identifier to identify things. For example, you can have a singular issuer with multiple keys - PKI can get really complicated and getting everything right is really difficult - as seen by the dozen implementations getting it wrong.
From my trips to the source code, I believe OpenSSL to be mostly certificate based.
Also interesting thought: Most modern implementations allow you to put not self signed certificates (e.g intermediates) in the trust store and assign trust to them [even if the actual issuer is unknown and untrusted] - does this make them roots or intermediates?
Well, public/private keypair is probably the correct term, as one can't exist without the other in RSA/ECDSA (and probably for EdDSA too?). I mean, they are mathematically bound to each other.
True, the Subject is referring to the keypair indeed, as per RFC 5280:
The subject field identifies the entity associated with the public key stored in the subject public key field.
Assuming the "entity" referred there is the keypair.
IMO roots, as they can be used as an anchor for a trusted certificate chain. And makes sense if we agree a trust anchor is an entity (i.e.: keypair) and not the certificate itself. So who cares who signed the cert
) So I can correct it if I run 
into again.
