Certbot returns certificates with the old certificate chain

Hi

Due to the recent trust chain changes we created new certificates for our staging environment in order to test these. We have in total four different certificates in that environment. When we got the new certificates, two of them had the new trust chain but two of them still had the old one!

I looked on crt.sh for the new certificates and the certificates listed there use the new correct trust chain. But the ones we received via certbot are not the ones that are listed there! The signatures does not match.

We're using certbot with certonly and providing a presigned csr, due to legacy reasons. If that makes any difference.

Best regards
//Johan

1 Like

When you say "received via certbot", do you mean that some fullchain.pem files have 3 certificates, while others only have 2? That is, how exactly are you testing? (The reason I ask is that web browsers will do everything they can to figure out what chains might work, so they don't actually report on what the web servers are sending. You need to connect to the server with OpenSSL or SSL Labs or the like, or look at the fullchain.pem being served, to tell what chain a server is actually using.)

2 Likes

For anyone looking at this topic who doesn't know how to do the test that @petercooperjr mentioned, it looks like

openssl s_client -connect example.com:443 -servername example.com

or, to choose an example that is actually secured with Let's Encrypt

openssl s_client -connect community.letsencrypt.org:443 -servername community.letsencrypt.org

The chain is presented at the very beginning of the voluminous output from this command.

For example, my output (near the top) looks like

Certificate chain
 0 s:CN = community.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

followed by many other details. This shows that the server is sending two certificates in its chain, and identifies which they are. Here s: means "subject" (what entity or site the certificate is for) and i: means "issuer" (who signed the certificate).

Probably a lot of people are going to want to know how to do this in the near future as a result of the chain changes!

The SSL Labs test is an online test which you would access by going to

https://www.ssllabs.com/ssltest/

and typing in your domain name.

4 Likes

@petercooperjr. That is exactly what I am saying. The fullchain.pem files for two of the domains have 3 certificates, while the other two has 2 certificates. I have the fullchain.pem files locally and have verified them by just opening them with a text editor. I have also opened them with Keystore explorer explorer and inspected the chain there. Lastly I have looked at the chain.pem with openssl x509 -in chain.pem -noout -issuer and have gotten these outputs:
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
from the ones with the longer chain
and:
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
from the ones with the shorter chain.

1 Like

Welcome to the Let's Encrypt Community, Johan :slightly_smiling_face:

Might you share the serial numbers of these four certificates? I can use them to cross-reference with https://crt.sh to match their times of issuance against the Boulder CA update.

Sure. Here they come:
Old chain:
03:f5:6a:12:a7:ec:1e:8e:9e:03:7e:0a:f7:7c:82:0d:58:46 (crt.sh | 4477626450)
03:b8:e4:b4:cb:54:4e:55:73:ad:ee:86:8e:19:d5:b4:b0:82 (crt.sh | 4477623136)

New chain:
04:e7:1d:bf:f7:73:df:37:fd:43:27:46:5e:02:c5:c3:87:45 (crt.sh | 4477615156)
04:38:d7:68:8e:21:48:d8:51:0d:15:57:f1:1e:47:a7:2b:9d (crt.sh | 4477619583)

BR
/Johan

1 Like

So, interestingly the two old chains were issued minutes after the two new chains. To the best of my knowledge, all four chains should have been the same chain structure based on the update timings. Unfortunately, I could find no history on crt.sh to know which certificates were sent by Boulder during the issuance process as there was no apparent indication which version of the R3 intermediate (signed by DST Root CA X3 or ISRG Root X1) was used to sign the certificates. The situation is very peculiar indeed as there is no logical reason that Boulder would be issuing the new chain and then somehow revert to issuing the old chain minutes later without some time of temporary fallback occurring (unless there were multiple Let's Encrypt servers operating under a rolling restart and your later certificates somehow were issued by unrestarted servers while the earlier certificates were issued by restated servers).

I've tried to get a new chain cert a few times recently and apparently failed, which I have to investigate (probably my bug). I was wondering how the rollout was being conducted - are some API servers issuing the new default chain and the rest are issuing the old chain? That would make sense for a phased rollout but it's hard to know what you're going to get.

1 Like

FWIW, when I updated the code for CertSage recently for compatibility with the new chain, all of the chains that I received in testing were the long chain.

1 Like

I was wrong in my assertion earlier, when I said that the signatures does not match. Apparently I looked at the precertificate and not the leaf certificate. The signatures on https://crt.sh and the signatures on the certificates I have are the same for all certificates.

1 Like

When I was first made aware of this issue, I figured that I had botched the renewal somehow. I therefore got new certificates the day after, but still got the old chain.
Here are the serial numbers for those:
03:cd:e9:ac:2a:c1:98:88:db:c4:61:35:d2:96:af:4f:17:ac (crt.sh | 4481836060)
03:43:97:64:31:54:c9:bd:ef:19:0e:a2:ce:e5:8a:3b:d0:db (crt.sh | 4481838346)

1 Like

That's even more surprising. Surely any type of rolling restart would have completed almost 20 hours later.

I'm looking at the serial numbers issued for the certificates. All the certificates which have the old chain have a serial number starting with 03, and all the certificates with the new chain have a serial number starting with 04. Coincidence? I don't know what the serial numbers that Let's Encrypt assigns are based on.

1 Like

Yeah Let's Encrypt does use prefixes in the serial number. Don't know for sure how they work though.

2 Likes

I looked into the Boulder source code to see where the serial number prefix comes from. From what I can tell it comes from a config struct which is in turn loaded from a config file. In my quick overview I found no place where this config struct is modified in any way in a running application. Therefore it leads me to believe that the certificates were issued by (at least) two different server applications which have different configurations.

2 Likes

I just checked the certificates issued for certsage.com in February (before the chain switch) and May (after the chain switch). Both certificates issued in February have a 04 prefix. Of the two certificates issued May 5th, the first certificate issued had a 03 prefix while the second certificate issued had a 04 prefix. The certificate issued May 8th had a 03 prefix. I'm fairly certain that I inspected all three certificates issued in May while in the process of updating the code for CertSage for the chain switch. If there is/was a correlation between the prefix and chain, I'm not able to provide any evidence beyond that I can say for sure that the certificate issued on May 8th with a 03 prefix is the long chain.

Really? According to SSL labs your certificate also use the short chain.
https://www.ssllabs.com/ssltest/analyze.html?d=certsage.com&latest

1 Like

That's probably just the GoDaddy cPanel grabbing the R3 intermediate certificate signed by DST Root CA X3 from a repository. When a certificate is installed via the webpage in cPanel, only the leaf certificate is pasted, not the intermediate certificate, which is then later automatically filled by cPanel. Here is the actual certificate chain sent by Boulder without any adulteration (straight from the ACME download link from Boulder at issuance to the file below):

certificates.txt (5.5 KB)

I think you made the same mistake as I did when I first verified against https://crt.sh. You're looking at the precertificate, not the leaf certificate. Your leaf certificate is located here: crt.sh | 4490725780, and looks like the one in certificates.txt

1 Like

You are correct. :man_facepalming:

I trusted that the deduplicate option in crt.sh would remove the precertificates from the list like it's supposed to do. Obviously it didn't here.