Due to the recent trust chain changes we created new certificates for our staging environment in order to test these. We have in total four different certificates in that environment. When we got the new certificates, two of them had the new trust chain but two of them still had the old one!
I looked on crt.sh for the new certificates and the certificates listed there use the new correct trust chain. But the ones we received via certbot are not the ones that are listed there! The signatures does not match.
We're using certbot with certonly and providing a presigned csr, due to legacy reasons. If that makes any difference.
When you say "received via certbot", do you mean that some fullchain.pem files have 3 certificates, while others only have 2? That is, how exactly are you testing? (The reason I ask is that web browsers will do everything they can to figure out what chains might work, so they don't actually report on what the web servers are sending. You need to connect to the server with OpenSSL or SSL Labs or the like, or look at the fullchain.pem being served, to tell what chain a server is actually using.)
The chain is presented at the very beginning of the voluminous output from this command.
For example, my output (near the top) looks like
Certificate chain
0 s:CN = community.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:O = Digital Signature Trust Co., CN = DST Root CA X3
followed by many other details. This shows that the server is sending two certificates in its chain, and identifies which they are. Here s: means "subject" (what entity or site the certificate is for) and i: means "issuer" (who signed the certificate).
Probably a lot of people are going to want to know how to do this in the near future as a result of the chain changes!
The SSL Labs test is an online test which you would access by going to
@petercooperjr. That is exactly what I am saying. The fullchain.pem files for two of the domains have 3 certificates, while the other two has 2 certificates. I have the fullchain.pem files locally and have verified them by just opening them with a text editor. I have also opened them with Keystore explorer explorer and inspected the chain there. Lastly I have looked at the chain.pem with openssl x509 -in chain.pem -noout -issuer and have gotten these outputs: issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
from the ones with the longer chain
and: issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
from the ones with the shorter chain.
Might you share the serial numbers of these four certificates? I can use them to cross-reference with https://crt.sh to match their times of issuance against the Boulder CA update.
Sure. Here they come:
Old chain: 03:f5:6a:12:a7:ec:1e:8e:9e:03:7e:0a:f7:7c:82:0d:58:46 (crt.sh | 4477626450) 03:b8:e4:b4:cb:54:4e:55:73:ad:ee:86:8e:19:d5:b4:b0:82 (crt.sh | 4477623136)
New chain: 04:e7:1d:bf:f7:73:df:37:fd:43:27:46:5e:02:c5:c3:87:45 (crt.sh | 4477615156) 04:38:d7:68:8e:21:48:d8:51:0d:15:57:f1:1e:47:a7:2b:9d (crt.sh | 4477619583)
So, interestingly the two old chains were issued minutes after the two new chains. To the best of my knowledge, all four chains should have been the same chain structure based on the update timings. Unfortunately, I could find no history on crt.sh to know which certificates were sent by Boulder during the issuance process as there was no apparent indication which version of the R3 intermediate (signed by DST Root CA X3 or ISRG Root X1) was used to sign the certificates. The situation is very peculiar indeed as there is no logical reason that Boulder would be issuing the new chain and then somehow revert to issuing the old chain minutes later without some time of temporary fallback occurring (unless there were multiple Let's Encrypt servers operating under a rolling restart and your later certificates somehow were issued by unrestarted servers while the earlier certificates were issued by restated servers).
I've tried to get a new chain cert a few times recently and apparently failed, which I have to investigate (probably my bug). I was wondering how the rollout was being conducted - are some API servers issuing the new default chain and the rest are issuing the old chain? That would make sense for a phased rollout but it's hard to know what you're going to get.
FWIW, when I updated the code for CertSage recently for compatibility with the new chain, all of the chains that I received in testing were the long chain.
I was wrong in my assertion earlier, when I said that the signatures does not match. Apparently I looked at the precertificate and not the leaf certificate. The signatures on https://crt.sh and the signatures on the certificates I have are the same for all certificates.
When I was first made aware of this issue, I figured that I had botched the renewal somehow. I therefore got new certificates the day after, but still got the old chain.
Here are the serial numbers for those: 03:cd:e9:ac:2a:c1:98:88:db:c4:61:35:d2:96:af:4f:17:ac (crt.sh | 4481836060) 03:43:97:64:31:54:c9:bd:ef:19:0e:a2:ce:e5:8a:3b:d0:db (crt.sh | 4481838346)
I'm looking at the serial numbers issued for the certificates. All the certificates which have the old chain have a serial number starting with 03, and all the certificates with the new chain have a serial number starting with 04. Coincidence? I don't know what the serial numbers that Let's Encrypt assigns are based on.
I looked into the Boulder source code to see where the serial number prefix comes from. From what I can tell it comes from a config struct which is in turn loaded from a config file. In my quick overview I found no place where this config struct is modified in any way in a running application. Therefore it leads me to believe that the certificates were issued by (at least) two different server applications which have different configurations.
I just checked the certificates issued for certsage.com in February (before the chain switch) and May (after the chain switch). Both certificates issued in February have a 04 prefix. Of the two certificates issued May 5th, the first certificate issued had a 03 prefix while the second certificate issued had a 04 prefix. The certificate issued May 8th had a 03 prefix. I'm fairly certain that I inspected all three certificates issued in May while in the process of updating the code for CertSage for the chain switch. If there is/was a correlation between the prefix and chain, I'm not able to provide any evidence beyond that I can say for sure that the certificate issued on May 8th with a 03 prefix is the long chain.
That's probably just the GoDaddy cPanel grabbing the R3 intermediate certificate signed by DST Root CA X3 from a repository. When a certificate is installed via the webpage in cPanel, only the leaf certificate is pasted, not the intermediate certificate, which is then later automatically filled by cPanel. Here is the actual certificate chain sent by Boulder without any adulteration (straight from the ACME download link from Boulder at issuance to the file below):
I think you made the same mistake as I did when I first verified against https://crt.sh. You're looking at the precertificate, not the leaf certificate. Your leaf certificate is located here: crt.sh | 4490725780, and looks like the one in certificates.txt
