The devil is in the details.
It is the same exact end-leaf cert.
The problem occurs when the wrong/expired intermediate cert is being served.
No intermediate is being served and the client tries to build the trust path (incorrectly).
The client doesn't listen to the intermediate being served and continues using the expired trust path.
The devil is in the details.
R3 is a keypair. There are multiple certificates for this keypair. Your "issued certificate" (we call it leaf) is issued by this keypair. All certificates called R3 are technically/cryptographically valid (for leaf validation).
So which R3 to use then? This is ultimatively decided by the client, but servers can send intermediates, together with the leaf. This gives the client a "hint" what it should use.
Your acme client should fetch the intermediate certificates automatically (on each regular renewal, usually placed into a "fullchain" file) and your webserver should be configured to serve them. If this is already the case, no action at all is neccessary.
If not, your server configuration may need to be corrected. All of this really depends on your exact setup: What webserver, what acme client etc.
Technically it is, unless software safeguards catch this. But this isn't what happened here, as the keypair was perfectly valid.
...EVERYTHING is possible...
I don't think any of the software used today allows for such an obvious problem.
While I hope that actual enterprise CA-software (like boulder) indeed catches this, smaller tools like xca or OpenSSL don't complain at all when you do such things. Issuing certificates that are valid beyond the lifetime of their issuer is also not that rare, e.g Let's Encrypts ISRG Root X1 cross-sign does exactly this. Other examples are Amazon's Server CA 1b, which is valid until 2040 - but the root that signed it (Amazon Root CA 1) is only valid until 2038, so the intermediate is valid much longer than the leaf. Will this work? No, but the certificates were issued anyway. They will probably be replaced before any actual issues arise. Granted, none of these cases involve leaf certificates, but technically it's the same thing.
What LetsEncrypt Client did you use to obtain the expired certificates?
As others have noted above, the R3 certificate expiring on 9/29/2021 was replaced earlier this year; the replacement is also called R3 but has a future expiry which is past the DST expiry (9/30/2021). Most browsers and devices will accept this, because they will short-circuit the validation to the ISRG-X1 root certificate that signed the identical keypair as R3.
Because of short-circuiting, how SSL clients build chains, and the presence of cross-signed certificates, devices/browsers will not reliably show the actual certificates used during a connection - they often just show (or use) the first valid certificates that match a key pairing.
So why do we care about LetsEncrypt clients?
Many clients have ignored the RFC and hardcoded the certificate chains, so when using them you will download a valid certificate from LetsEncrypt but it will be chained against a potentially invalid certificate bundled into the client's source code. This will still work in production for many situations if the keypairs are the same (as in R3-old and R3-new) and the client can build a valid path, but analyzing the chain locally can generate errors.
I'm having the same issue. my server is a Go REST api that uses the fullchain.pem file. I used the current certbot snap on ubuntu 18.04 to force-renew a new cert today. How do I get certbot to use the R3-new and not the R3-old? is this something that I need to do in certbot, or the cert listings in ubuntu, or do I need to use a different file instead of the "fullchain.pem" in my REST api server.
I saw this exact problem on my Mac. Whatever the cause (and I can't pinpoint it) the solution was to modify my server to use 'fullchain.pem' instead of 'cert.pem' for dovecot imap. That solved the problem for all my personal email domains on my server.
Hi @aparduhn, welcome to the LE community forum
certbot gets what LE provides.
[which has been the new R3 for quite some time now]
Although I would ask you to check what version of OpenSSL you are running.
Definitely NOT; that is the right file.
Are you sure it is using a recent version of it?
Hi all - I'm a new user and had an AWS expert set up my certs. Today, both of my websites, with current, valid certs, are showing as expired, and users are getting the "This connection is not private" warning message on Safari in iOS. Can anyone guide me here?
@BrianCanFixIT - how did you "force a renewal of the websites that were affected?"
What ACME client are you using?
Which of those cert renewal files are being used by your service?
Thank you so much for the quick response. I'm using OpenSSL 1.1.1 11 Sep 2018. and the version of certbot is installed: 1.19.0 (1434) 44MB classic
Certs are managed on AWS Certificate Manager. I'm not sure how to answer your 2nd question. (novice here)
Check to see that you're using 'fullchain.pem' - that solved all my Mac OS and IOS problems with no change on the client side.
Based on what you said, that sounds like my problem (issues are isolated to Mac OS and iOS users. Now, novice user here, where would I check for that and/or update to "fullchain.pem?"
If you can, test to see what is being served by your service.
Try something like:
openssl s_client -connect EXAMPLE.APP:443 -servername EXAMPLE.APP
If you are in a hurry/emergency state, I guess I would have to go to "in case of emergency break glass"...
And have you try to switch to another free CA until this problem is resolved.
Is is a publicly accessible website?
Yes - www.thecollegeagency.com
That depends on which softwarre. Apache? Ngnx? Dovecot? Postfix? Basically look for the configuration file of whatever software you're using and change 'cert.pem' to 'fullchain.pem' and you're good to go.