AWS with CentOS - tweaking configs

May not help everyone but after hours of head banging, This was my fix (on AWS EC2 linux) CentOS and Redhat should work as well.

  • Copy the DST X3 cert into /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem

In my case, that was:

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

then run:

sudo update-ca-trust

I also removed the X3 cert from my server's chain.pem file and restarted. everything works!

2 Likes

with AWS with CentOS here too! if you can help me please:

  • how did you know your DST Root CA X3 certificate? where it was located?
  • you remove it manually from chain.pem and fullchain.pem ?

i'm stuck, as https://chainchecker.certifytheweb.com/ says "Let's Encrypt Modern Chain" (i understand is fine) and my browser can see the page with no problems, BUT when I execute

openssl s_client -connect mysite:443

it says "certificate has expired" :pensive:

Please don't use that tool for this.
Try:
SSL Certificate Checker - Diagnostic Tool | DigiCert.com
OR
SSL Checker - Test Certificate and Installation (certlogik.com)
OR
openssl s_client -connect EXAMPLE.org:443 -servername EXAMPLE.org

Hi @punchi, welcome back! I see you've been around the forum for five years - since way back at the beginning. :smiley:

Can you please post your actual hostname? That will let us verify the certificate chain being served.

Also, please share the output of openssl version, and what version of CentOS you are on.

3 Likes

You may also find this thread useful: RHEL/CentOS 6 OpenSSL client compatibility after DST Root CA X3 expiration

3 Likes

thanks! but i don't understand the certlogik.com result... says:

Trusted -> No (The certificate has expired.)

Ok, I still have the problem, but in Certificates Received says,

Let's Encrypt Certificate 
Valid To 	29 Dec 2021, 8:16 p.m.
R3 Certificate 
Valid To 	15 Sep 2025, 4 p.m.
ISRG Root X1 Certificate 
Valid To 	30 Sep 2024, 6:14 p.m.

So, the certificate "is expired" but in the details no one is expired :no_mouth:

It's an imperfect tool for an imperfect world.
What do the others have to say?

Hi @punchi,
what's happening is that your certificate chain (or, the server's CA chains, still have the cross-signed X3 CA registered as an active CA). You need to manually disable the X3 CA so your server stops sending it out. In my case, I had an MQTT server that relied on the built-in CA chains in the operating system (AWS Linux). The MQTT server had both the full chain in it's cert file and the OS had the CA listed as active. I have no idea if that would change over time but in the short term, it was causing the MQTT server to send out the work root CA (X3).

The MQTT servers' .pem file had four certificates, the server cert, the R3 cert, the ISRG X1 cert, and the DST Root X3 cert. You can see the X3 cert in my original post. It would most likely suffice for you to copy and paste it into a file as I had mentioned in my earlier post and then update the ca trust file. I removed the cert from the .pem chain file as an extra precaution. If you are having this issue with a web server, the certificate location will be listed in the config file (typically /etc/nginx/sites/available/sitename.com for Nginx and /etc/httpd/sites-available/sitename.conf or /etc/apache2/sites-available/sitename.conf for Apache although it will vary by installation).

you could also do a

sudo find /etc -type f -name "*.pem"

or

sudo find /etc -type f -name "*.crt"

to locate any .crt or .pem files in /etc

Let me know if you are still stuck and the configuration you are trying to get working again and I'll see what help I can be.

3 Likes

wow! lot of replies! I think I got it fixed =)

sudo yum update

a new "ca-certificates" package have arrived! (For Amazon Linux 1 & 2)
can you validate it's fine @rg305 please? app.gesnex.com and www.gesnex.com

thanks for everyone!

looks like I was not enough :pensive:

@jsha @frsp1 @rg305 anyone knows how to rip off the Path #2 ?? there's no DST Root CA X3 here /etc/letsencrypt/live/gesnex.com/fullchain.pem where it can be located this path 2? :grimacing:

thanks!!

Let me explain that a bit (there is no way to remove it - a valid path will always be known/found).
The last cert in each always ends with "in trust store".
Those certs should never be sent by a server - they are to be found in the root stores that we all trust.
So,, what's the difference between path #1 and #2? They both start exactly the same:

  • Certs 1 & 2 are identical in both paths (same serial #).
  • but cert 3 is different (same name but different serial numbers)
    Path #1 cert 3 is in the root store (so it will always be known)
    Path #2 cert 3 is NOT in the root store and is known (so it must have been sent by the server)
    It is that cert that ties the chain to the EXPIRED root cert.

If you want to "break" that path, you need only remove one link from that chain.
1 & 2 must be sent to complete path #1 and 4 is already in the root trust store...
So that only leaves the removal of 3.

If you want to "break" path #2, then don't send path #2 cert 3 "ISRG Root X1 (cross-signed)".
Without the hints, only systems that have cached it would likely ever try to use it.
[thus the need to delete it from stores and even memory (reboots are sometimes required)]

4 Likes

@punchi Check your openssl version. If it is 1.0.2k like in AWS Linux 1 and 2 then that version needs the -trusted_first option so it behaves like openssl version 1.1.

Example:

openssl s_client -connect community.letsencrypt.org:443 -servername community.letsencrypt.org -trusted_first

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = community.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:/CN=community.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Without -trusted_first, a chain with DST Root CA X3 will cause openssl to show an expiration error like this:

depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT

See this openssl blog for more details
https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

4 Likes

thanks for the nice explanation @rg305 !! :grin: but now how can I stop sending the ISRG Root X1, fingerprint SHA256: 6d99fb265eb1c5b3744765fcbc648f3cd8e1bffafdc4c2f99b9d47cf7ff1c24f ?? where is located? :pensive:

3 Likes

That depends on the ACME client used.
Usually with some form of fullchain file.
The simplest/fastest way is to find that current file and manually edit it - remove the last cert from within it and then restart your web service.
NOTE: This "workaround" (not permanent fix) will last until the next renewal (automatic replacement of that fullchain file).

5 Likes

thanks!! it worked :grin:

2 Likes

Great!
The next step is to make that change permanent.
How, depends on which ACME client you are using (and version).

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.