I'm trying to do some testing on older devices ahead of the September expiration of the DST root, but for the life of me, I can't find a way to issue a certificate signed only by ISRG as it would be come September. Is that possible yet?
2 Likes
The end entity / leaf certificate is the same. The only difference is the chain. Some clients allow you to specify an "alternate chain selection". If yours does not, you can just download the chain from here: Chain of Trust - Let's Encrypt
Edit: The way LE leveraged the DST root was by having their Root Cert cross-signed by DST. The fullchain paths are either :
- Cert -> R3 -> ISRG Root X1 Cross Signed by DST -> [DST in Device Trust Store]
- Cert -> R3 -> [ISRG Root X1 in Device Trust Store]
2 Likes
Thanks. I don't think I'm following you. I'm using Posh-ACME which does support the preferred-chain parameter. But even with preferred chain, I am getting:
Cert -> R3 -> ISRG Root X1 Cross Signed by DST -> [DST in Device Trust Store]
I believe that's what I am getting because when I inspect the certificate, I'm getting this:
I guess what I am trying to do is get an R3 Intermediate not cross signed by IdenTrust's DST X3. Am I asking that properly / making sense.
1 Like
Don't use your browser as a method to check the chain send by the server: browsers can ignore the send chain and build their own.
Please use other methods of figuring out the chain such as sites like https://whatsmychaincert.com/ Uch, that site doesn't even let you look at the chain, it just says it's OK. Well, that was NOT what the sites URL advertised! 
Anyway, sites like SSLLabs also show the chain send by the server: SSL Server Test (Powered by Qualys SSL Labs)
1 Like
Yes. You can just download that off the LetsEncrypt Certificates page I linked to above and install that on your server.
Trying to better explain what @Osiris wrote: Browsers and OpenSSL libraries all build chains however they see fit, and are not reliable ways to see what the server used. You'll need to use a tool like they linked to, to ensure your server is actually serving the chain you want.
Some browsers and openssl libraries just look at the cert data, and see if the signing information chains to something in it's recent cache. Others will loop the trust store in order of chain length. Some will loop in the list's order. Others probably alphabetically on something. It's a common point of confusion, because a browser or device will show the trust path roots up to DST when X1 was specified by the server, or X1 when DST was specified by the server. This is because the keys on both are the same. The browser/library is just showing you the trust path it built - not the one specified.
2 Likes
Note that the openssl s_client -connect command actually does show the actual chain send by the server and utters a warning if the chain is incorrect.
In this case the server of libbyalpha.com is just sending the end leaf certificate without any intermediate certificate. This can lead to certificate issues in some browsers and intermittend issues with others, depending on whether that browser has cached an intermediate cert it can use to build a valid chain.
2 Likes
Thanks all so far. @Osiris , yea, the incomplete chain was a bad test on my part. We're deploying what I hope is the proper cert now. Will test with SSL Labs once it's deployed.
1 Like
I'll remember that. I should have said SSL libraries. There are a lot of different ssl libraries - either language bindings for the openssl project or alternate implementations. They are all over the place in terms of functionality, even amongst the openssl bindings.
2 Likes
@jvanasco Using OpenSSL from the command line with s_client could have different behaviour than using OpenSSL as a SSL library too perhaps, I dunno. Just wanting to prevent any misunderstanding if people see the openssl s_client command used and don't trust its results 
1 Like
Ok. So I manually downloaded the R3 intermediate signed by ISRG X1
Combined them and uploaded. SSL Labs still shows both paths.
Am I right in assuming that is because SSL Labs knows R3 is cross signed even though I didn't provide it? Unfortunately, it seems my Apple iOS9 devices are doing the same because the certificate still works on them. But according to what I'm reading, in September, iOS 9 will start to fail. End of the day, that's what I am trying to simulate now. Am I missing something?
1 Like
Note the "extra download" stated on the second path. That means this chain wasn't send by the server, but a client could build this chain, if it wanted too (and is capable of grabbing that intermediate from somewhere).
The path building logic on SSL Labs is not too smart and doesn't always display every possible verification path.
3 Likes
If you can, and you have certificates which expire in September or later so that they'd still be valid when the DST certificate expires, you should try to simulate the future date in iOS to see what actually happens. I cannot advise how best to achieve this, or even whether it's possible, but one would think naively that turning off automatic date-time and manually setting September 2021 could work.
This was trickier months ago because you couldn't get a leaf certificate for your server that hadn't expired by September, and so you'd get an error either way. But now of course the certificates are valid well into September, so if your iOS 9 device can be convinced it's already September you can see what will happen.
1 Like
Unfortunately, setting the time on our iOS test devices didn't work. It caused a slew of other errors. Good idea though.
Digging a bit more, I am starting to think my failure is that my preferred-chain parameter didn't actually work. When I generated the certificate, I used "ISRG Root X1" as the preferred-chain. Then, testing with:
openssl s_client -servername libbyalpha.com -showcerts -connect libbyapp.com:443
I see I'm providing the longer chain that is signed by DST.
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.libbyapp.com
verify return:1
So based on that, I think the attempt to generate a cert, using preferred-chain='ISRG Root X1' didn't actually give me a the alternate chain and instead gave me the longer chain with the ISRG Root cross signed by DST. So with that said, is anyone aware of what value I should have used in the preferred-chain parameter to get the ISRG Root not cross signed by DST?
1 Like
According to the Posh-ACME documentation, -PreferredChain 'ISRG Root X1' is correct.
@rmbolger might have an idea.
1 Like
Looks to me as though you've been working on this as I was looking at it, since the answers I got varied 
The configuration I was last seeing in which you server presents the certificate for *.libbyapp.com (signed by R3) and then the certificate for R3 (signed by ISRG X1) which I think is what you intended.
1 Like
I was working through it a bunch yesterday, but haven't touched it yet today. I thought I did it right, but testing with our iOS 9 devices, which I would expect to fail because iOS 9 does not trust ISRG, succeed. That suggests to me that somehow the DST root is still involved.
2 Likes
Keeping in mind that the RSA chains currently being issued are as follows:
default chain
sent: leaf certificate ← R3 ← ISRG Root X1
expected in trust store: DST Root CA X3
alternate chain
sent: leaf certificate ← R3
expected in trust store: ISRG Root X1
It can readily be seen that both chains contain ISRG Root X1. Therefore, I'm not sure that specifying 'ISRG Root X1' will accomplish anything.
That is indeed correct for getting the non DST chain. There are a couple ways you can verify.
If you look at the order folder in Explorer, there should be chain.pem, chain0.pem, and chain1.pem. The first one is the "active" chain file used to build the rest of the combo files. It should show as 2 KB file size if it's the ISRG Root X1 chain. The DST chain is 4 KB. You can also open the file and it should only have one cert in it whereas the DST chain will have two.
For reference, the other two chain files are cached copies of all chains offered by the CA. So chain.pem is a duplicate of either chain0.pem or chain1.pem. They allow you to change which chain you're using with Set-PAOrder later.
2 Likes
Glad to see that this is working in Posh-ACME. I know it had caused some headaches elsewhere for the reason I stated above. 
It's a bit goofy that one can simply strip off the last "intermediate" certificate from the default chain in order to "build" the alternate chain.