Our Amazon AMIs took a dive a few hours ago because the older Root X3 cert was still at:
cd /usr/share/pki/ca-trust-source/ca-bundle.trust.crt
This is an Amazon instance, I didn't even consider an auto-updating instance that I would need to do anything. I had to surgically remove the Root X3 cert from ca-bundle.trust.crt and run update-ca-trusts enable && update-ca-trucks extract to resolve.
This seems like overkill. Did I miss an easier option?
We have problem when our application try to connect to our clients from AWS servers. all of them
have let's encrypt. i tried to update root certificates manualy but it didn't work. Please help me to solve this problem
Amazon updated their ca-certificates bundle on Sept 30th at 8.25PM UTC. This update removes the Root X3 certificate, so you did not miss anything.
Hi! I'm in trouble with a Amazon instance with the same problem. Which part of ca-bundle.trust.crt did you remove? Could you show me?
That is correct. It is resolved anyway now since Amazon released a new ca-certificates package that resolves it.
ca-certificates noarch 2018.2.22-65.1.24.amzn1 amzn-updates 1.0 M
That manual fix was needed if you needed to resolve the issue in the 9~ hours it took Amazon to release an official update. It should not be needed at this time.
Hi @rg305. This was what I'm thinking, but in my ca-bundle.trust.crt file there is no mention to this cert. When I check chain with openssl this is the result:
Certificate chain
0 s:/CN=roupasparaciclismo.com
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
My Amazon AMI instance uses Fedora with PHP 5.6. When I do:
update-ca-trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
I got:
unable to load certificate
140547583727440:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: TRUSTED CERTIFICATE
Could you help me please?
@hallandellon
Where did you get that instruction?
Does it fail similarly with just the first part?:
update-ca-trust dump --filter \
"pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10"
I get the same result on:
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.31.1.el7.x86_64
Architecture: x86-64
update-ca-trust dump --filter "pkcs11:id=%c4%a7%b1%a4%7b%2c%71%fa%db%e1%4b%90%75%ff%c4%15%60%85%89%10" | openssl x509 | sudo tee /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
unable to load certificate
139676756105024:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
I used the instructions posted here: Correct expired Let’s Encrypt certificate on EC2 instances. But the trust command is not recognized by the system and I could not install it. What's the right way to do?
You get that because the operating system probably already updated ca-certificates package and removed the DST Root CA X3 expired cert. As it is already removed, update-ca-trust dump will not find it. In otherwords, you should already have the the expired removed via ca-certificates operating system updates as explained at Correct expired Let’s Encrypt certificate on EC2 instances | AWS re:Post.
In the ca-bundle.trust.crt really doesn't show the DST Root CA X3 as explained before. But why the chain still shows the cert?
Certificate chain
0 s:/CN=roupasparaciclismo.com
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
I'm having problem to connect through an api in this server due to this.
Hi @iBotPeaches, and welcome to the forum! For the benefit of others, would you mind listing the AMIs that had this problem?
In general, the solution you offer here, surgically removing DST Root CA X3, is a quick fix. The better fix is to upgrade to a supported version of OpenSSL (1.1.1). It would be great to reach out to the maintainers of the affected AMIs and get them to upgrade OpenSSL.
Hi @jsha , I had two AMIs affected by this
The surgical fix resolved all our errors that popped up immediately moments after the expiration. It unfortunately took me 49 minutes though to solve that and then even longer to release that patched AMI to the rest of the fleet. I was not aware that the OpenSSL fix (though I hope AWS would keep that package up to date) was included. As both of those AMIs had a simple "yum update" ran weeks prior.
Is the updated OpenSSL the fix that caused an expired cert in my trust store to be leveraged despite the newer ISRG Root X1 in the store? I kept trying to research what logic does that and I found hints that it uses the subject line to depict a newer certificate from an older (expired) one.
Does that align with your thinking? Since it does line up with the subject difference
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3Hi @hallandellon, welcome to the LE community forum 
If I read that correctly, he had to edit the file and remove the section for "DST Root CA X3" from it.
I found a solution following what @iBotPeaches said, but without use the "update-ca-trust enable && update-ca-trust extract". The .crt files was in /etc/pki/tls/certs/ folder. So, I just removed the "DST Root CA X3" part, and the API could communicate again. 
@iBotPeaches and @rg305
I started a VM of this type, logged in, and checked the OpenSSL version:
$ openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
Per /policies/releasestrat.html, 1.0.2 is no longer supported. Would you mind filing a support ticket with Amazon to upgrade the OpenSSL version in their AMI images?
Yes, or rather the other way around. Since your system has ISRG Root X1 in its trust store, OpenSSL should simply ignore the cross-signed version of it (signed by DST Root CA X3) that shows up in certificate chains. However, OpenSSL 1.0.2 doesn't properly ignore the cross-signature. Instead it throws an error! That behavior is fixed in OpenSSL 1.1.0.
Please read more about workarounds for this at Old Let's Encrypt Root Certificate Expiration and OpenSSL 1.0.2 - OpenSSL Blog
@jsha I am far from an expert but this has a long and complex history. Perhaps @petercooperjr can add more as he was a poster in one of the threads below. In short, AWS backports some CVE's without changing the version number so it is not as vulnerable as it appears. But, it also does not gain the later functions.
This post by an AWS rep in 2020 says they had no plans to update the system openssl and suggested how to upgrade and link other software (like nginx) to a newer version:
Some random related posts (the first has comments by Peter):
There is no harm in submitting more requests. It just does not seem likely they will soon change the built-in system version - although newer versions could be used with manual effort today. Providing details beyond what I regurgitated here is beyond my knowledge base
Hope you (or someone) find this helpful.
Ami 1/2 images are based on Redhat 6/7 RPM/DNF packages and they are more conservative and generally don't upgrade openssl beyond their 1.0.1/1.0.2 respective default versions. Redhat 8 defaults to openssl 1.1.1
I don't know if I have a whole lot more to contribute, but I'll see what I can do.
Amazon Linux (1) is roughly the same as Red Hat/CentOS 6.
Amazon Linux 2 is roughly the same as Red Hat/CentOS 7.
The Red Hat philosophy (also used by "clones" like Amazon Linux), as best as I can tell, is that as an "enterprise" system stability is more important than features. Security patches are backported, but that's about it. (And they don't update the "main" version number generally, so it can be a bit of effort to track down exactly which patches are installed.) I think the idea is that if you want new features, you can upgrade to a newer version of the OS, and do so in a controlled tested manner (as one would update your dev/test environments first, and so forth), and "patching" should be as minimally-invasive as possible because any change has a chance to be incompatible with what you're running on the server.
Amazon Linux 2 offers a bit more flexibility in that they offer an "extras library" of more up-to-date things you can install, but of course when you update more often you're trading off getting less stability.
For openssl specifically, they do now have an "openssl11" package for Amazon Linux 2 (in the "normal" repo, not the "extras") which will install the library and binaries. You could then run "openssl11" instead of "openssl" to run the updated version. But nothing "built-in" links to it, so my understanding is that if you wanted to use nginx or apache or whatnot you'd have you recompile it against the openssl11 library yourself. (So for instance, for my personal mail server, I compile dovecot & postfix from sources against it.)
Extended (paid) support for Openssl 1.0.2 is still available, and my understanding is that Amazon (and/or Red Hat/CentOS) get such support and still backport security patches. So, for instance, you'll see openssl (and openssl11) patches in Amazon's Security Advisory list.
For Amazon Linux 2, in the "Long Term Support" section of the FAQ, they say that they will provide "security updates and bug fixes for all packages in core until June 30, 2023."
For Amazon Linux 1, while it's "end-of-life" their statement says that they have a reduced set of packages that they'll be supporting until June 30, 2023, including "Low-level system libraries such as glibc and openssl."
So I think anyone using Amazon Linux's openssl is "secure" in the sense of getting security patches (at least for the next year-and-three-quarters), but there doesn't look to be any easy way to get "system" packages using openssl 1.1.1, there's just a way to have the openssl11 library and binaries installed.
I had been assuming for quite some time that at some point, Amazon might release an "Amazon Linux 3" based on Red Hat / CentOS 8 (or some future version), especially as their OS that specifically is called out as having "Long Term Support" now only has less than two years left in promised support. But I haven't seen any such announcement or roadmap (though it may be out there, I've not spent much effort looking for it). And there's been a lot of turmoil in the Red Hat / CentOS world this year, so perhaps Amazon is planning something else entirely.
In the meantime, if sticking with Amazon Linux one needs to deal with having only OpenSSL 1.0.2 available. OpenSSL has announced that they are working on version 1.0.2zb that "will make it possible to build the release with added -DOPENSSL_TRUSTED_FIRST_DEFAULT on the build configuration command line. That will make the -trusted_first option enabled by default by the OpenSSL library." So, when they release that, if Amazon (or Red Hat / CentOS, I don't know where the decision would originate) decides to backport that fix, then the version used on Amazon Linux would have that "trusted first" logic that allows for more easily connecting to servers with the "default" chain rooted in DST Root CA X3. That's a big "if" there, as I don't know if the change would fit the "security patch" criteria they use to decide to backport patches, and in any event won't help for the immediate future.
So in the meantime, you'll have to find another workaround. Let's Encrypt made an intentional choice to prioritize old-Android compatibility over old-openssl compatibility when deciding what would be the default chain, and perhaps if they had decided the other way around just as many people would be just as unhappy (though it may have just been a different set of people).