One of these certs is not like the others -DST Root CA X3 breaking antique hosts

Help
1

(forgive the anonymity, I read the faq but I don't want to publicise my customer's security holes)

I've successfully installed Let's Encrypt via acme.sh on multiple hosts at a customer site which are running Centos 6 and OpenSSL 1.0.1e-fips and Apache 2.2.15 (*) via these commands:

/root/.acme.sh/acme.sh --issue -d host.example.com --webroot /www/root/dinosaurapp --server letsencrypttest --staging --debug
/root/.acme.sh/acme.sh --issue -d host.example.com --webroot /www/root/dinosaurapp --server letsencrypt --force (once above was working)

for most websites we see something like the below:
openssl s_client -showcerts -connect foo.example.com:443

0 s:CN = foo.example.com
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Nov 11 23:19:27 2023 GMT; NotAfter: Feb 9 23:19:26 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT

But one and only one website has the extra entry #2 below. The other Centos6 hosts are failing to connect to this one because of the expired entry in the chain, which I believe is the old cross-signed cert:

openssl s_client -showcerts -connect bar.example.com:443

0 s:CN = bar.example.com
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
v:NotBefore: Nov 23 02:39:46 2023 GMT; NotAfter: Feb 21 02:39:45 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT

What could I have done differently here to get this different outcome, and how do we remove that last insecure entry? To add to the challenge, none of the (supposedly) identical development hosts picked up this entry in their chains.

The only other difference I do see in this one is the smaller key

The different host also had this slightly different Apache conf :
SSLCertificateFile /root/.acme.sh/bar.example.com_ecc/bar.example.com.cer
SSLCertificateKeyFile /root/.acme.sh/bar.example.com_ecc/bar.example.com.key
SSLCertificateChainFile /root/.acme.sh/bar.example.com_ecc/fullchain.cer
SSLCACertificateFile /root/.acme.sh/bar.example.com_ecc/ca.cer

however changing it to match the others had no effect:
SSLCertificateFile /root/.acme.sh/bar.example.com_ecc/bar.example.com.cer
SSLCertificateKeyFile /root/.acme.sh/bar.example.com_ecc/bar.example.com.key
SSLCACertificateFile /root/.acme.sh/bar.example.com_ecc/fullchain.cer

I'm slogging through trying to get the old cert out of the root store on the Centos6 boxes, or fake the expiration date, as discussed elsewhere on this site, but I'd rather get the cert out of the chain for fear that touching openssl might break the antique business application software.

Thank you for any clues on how to keep this breathing until it can get a brain transplant. Worst case is we buy a new commercial cert but we'd really like to make this work.

(*) The are aware of how hideously ancient this all is. However, upgrading the OS is blocked pending other efforts to port, upgrade, or replace the even more obsolete application software, which depends on obsolete packages, and so on. Meanwhile, I don't want to be the one to topple the Jenga blocks.

2

PS on a related note I've been poring over all of the comments in these closed threads

And a couple of q's -

  1. I'd been considering grabbing certificates.txt from a newer distro - why does this also require building 1.0.2?
  2. the advice below seems to also apply to Centos6 - at least I do have an update-ca-trust program which knows about that directory . I have the cert, in BEGIN/END form, which I can also convert to the longer format beginning wtih "Certificate" (not sure on the nomenclature)
    What sort of format does it need to be in, for the blacklist directory?

Put the expired CA certificate(s) in /etc/pki/ca-trust/source/blacklist/ , and invoke update-ca-trust(8) to rebuild the trust store. Verify with trust list .

3

Have you read this?

5 Likes
4

You probably don't need the last one that is only when using client authentication. In any case, that's not related to Let's Encrypt.

For your old Apache 2.2 you would use this method. Your names were somewhat odd but the essential part is the leaf cert is in first line and just chain (not fullchain) in the second:

SSLCertificateFile      (folder)/cert.pem
SSLCertificateChainFile (folder)/chain.pem
SSLCertificateKeyFile   (folder)/privkey.pem

This is odd. The default chain from Let's Encrypt includes the DST Root CA X3 so all should have it. But, you can configure acme.sh to choose the alternate shorter chain without that. Could you have those other names configured with a preferred-chain? I can't guess any other reason to see what you observe.

The one that fails is also using an ECDSA key which is why it is smaller. But, indicates the acme.sh config is somewhat different because your others are using the RSA key which was the old default and would be honored going forward unless specifically overridden.

In short, I think the acme.sh configs are different for these certs.

Is it required to have these old Centos6 servers connect to your others? Or is that just something you noticed during test? Because that's a very different issue than the above. And, why focus on the DST Root in the chain? Are you sure the ECDSA vs RSA key isn't causing the problem?

3 Likes
5

It looks like something has changed in the certificates served by LE. They used to include the intermediate ISRG Root X1 certificate, but have stopped doing so recently. See: Certificate chain no longer includes ISRG Root X1? - #3 by guillerodriguez

This could explain why you see different chains, depending on the time the certs were last renewed.

7

@guillerodriguez, to counter your point:
I've just used acme.sh v3.0.5 and v3.0.7 and received 3 certs [the long chain] from both.

To be clear, it incluced:

image
image405×512 9.04 KB

3 Likes
8

why windows think that certificate is revoked?

3 Likes
9

Interesting:

image
image406×515 8.81 KB

3 Likes
10

looks like crl.identrust.com is down: and windows certverify considers it revoked I think?

https://validation.identrust.com/crl/commercialrootca1.crl (their new root certificate's crl) is down too, so

is there regulation about CRL endpoint availability?

4 Likes
11

@rg305 Yes, my comment that "LE has stopped including ISRG Root X1" recently seems to be wrong. Sorry for that.

However the certificate with the short chain from the OP was renewed on Nov 12, which was within the time window where the LE servers were apparently misconfigured (Shortening the Let's Encrypt Chain of Trust - #2 by aarongable). So the fact that he is seeing different chains indeed depends on the time the certs were last renewed.

3 Likes
12

ahhh thank you that does explain why my dev box is not the same as my production box!

3 Likes
13

Pretty sure it was the DST root because I was seeing a cert expired error, and using the hack suggested elsewhere of changing the expiration date in the crt bundle with sed has mitigated the issue for the moment:

Haven't found config differences between servers, but noted below there was an issue with lets encrypt servers where they were serving the shorter chain for a time period , when most of my certs were either generated or renewed.

And yes, the old Centos 6 hosts have to connect to each other, as well as serving some pages to the general public . I've built some new hosts with current OS , but because of dependencies these three can't be lift-and-shifted at the app layer.

Am very relieved that in the short term everything is at least talking again - and to have an explanation for why the certs were different despite what should have been identical configurations.

1 Like
14

You should consider putting them behind a web proxy [that is more up-to-date].

4 Likes
15

Interesting idea. I will research it. Each of these antiques is a standalone SPOF , but a more modern frontend would be a big plus. They're all on the cloud so there are a lot of lego bricks available if we want to pay for them.

To make life even more interesting, the host 'bar.example.com' is also acting as a reverse proxy to pass calls from the other hosts back to a third-party service. I think this was done to prevent 'Cross-origin resource sharing" (a term I just learned last week) although I haven't figured out (yet) why this was necessary in this specific case. That's the piece that broke when the Centos6 boxes were rejecting the cert, taking a critical part of the workflow with it. I'd guess that if we moved the apps behind a proxy, these calls would also go to that proxy.

It might be also possible to move all of the calls to the third party service to this one host, which might eliminate the need for the reverse proxy (or not? there are a lot of pieces).

I've been a sysadmin for many decades but I'm new to web application support, have a lot to learn! I'm still trying to get the dev host, which I built, to work with the dev environment on the third-party service, which is blocking me from doing any serious reconfiguration.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.