On Windows, just reboot your server and the old R3 will disappear from your chain.
1 Like
Yes, I think that's the problem in our case. We are using a --deploy-hook script written in python and for some reason it doesn't update the intermediate cert. Well, it works now anyway with some hands-on magic. Have to look into it and see what's (not) going on.
Update: the script only copied privkey and cert, not fullchain. Problem solved.
The problem is that the last certificate in the fullchain generated by certbot shows
"version": 3,
"serialNumber": "4001772137D4E942B8EE76AA3C640AB7",
"notBefore": "Jan 20 19:14:03 2021 GMT",
"notAfter": "Sep 30 18:14:03 2024 GMT",
"caIssuers": [
"http://apps.identrust.com/roots/dstrootcax3.p7c"
],
"crlDistributionPoints": [
"http://crl.identrust.com/DSTROOTCAX3CRL.crl"
]
While NotAfter is Sep 30 18:14:03 2024 GMT, the issuer http://apps.identrust.com/roots/dstrootcax3.p7c is still serving the expired cert
How can I solve this?
My Windows/PC machine is able to access the site, chrome shows a chain where the R3 expires 2025.
But my iPhone only displays the old outdated R3 cert.
Other services that does webhooks against my server also fails, guess the expired R3 problem is at their end too?
Will a force renew of my SSL-certs for the affected domain solve this problem?
EDIT: Renewed the certs, rebooted iOS-devices, no go. This is a nightmare.
force renew won't change anything.
What is the site name?
Who manages the site/server?
1 Like
I manage the server.
The problem was that it's an old Debian 10 install, so the ca-certificate package is really old.
apt-cache show ca-certificates
Package: ca-certificates
Version: 20200601~deb10u2
I had to manually remove the DST_Root_CA_X3.crt and then restart the webserver.
I still have a hard time understanding how this is "supposed" to work on Debian.
As far as I can see, this is the latest package for stable/bullseye:
Debian -- Details of package ca-certificates in bullseye ( 20210119)
And that one still seem to have the old X3 cert:
#
# Certificate "DST Root CA X3"
#
# Issuer: CN=DST Root CA X3,O=Digital Signature Trust Co.
# Serial Number:44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
# Subject: CN=DST Root CA X3,O=Digital Signature Trust Co.
# Not Valid Before: Sat Sep 30 21:12:19 2000
# Not Valid After : Thu Sep 30 14:01:15 2021
There are two basic trust paths left for LE RSA certs:
-
one chains to "ISRG Root X1" which chains to "DST Root CA X3 (expired)"
This is really only used to provide service to old Android devices (they don't care if a root is expired) - it was thought to be the most compatible but has fallen a little short of expectations
[seems to be wreaking havoc on really old Windows systems and even to many iOS devices] -
one chains to a self-signed "ISRG Root X1"
This works for newer systems that have that cert in their root cert store
[or those that can be manually inserted therein]
2 Likes
well, I had to delete all certificates and generate without force-renewal. at least for certbot.
Good day.
I'm running postfix/dovecot and apache2 on Ubuntu 20.04.3 LTS.
All certificates are showing as expired.
I have done a forced refresh of all the certificates, but I'm still getting the same error.
I have checked the dates on the pen files, and they are all for today.
for apache2
openssl s_client -connect shortlinx.co.za:443
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
0 s:/CN=mail.lcsoftware.co.za
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
and for imap (postfix)
openssl s_client -connect office.lcsoftware.co.za:993
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
0 s:/CN=office.lcsoftware.co.za
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
I'm using certbot 0.40.0
my dovecot ssl configuration is
ssl = required
ssl_cert = </etc/letsencrypt/live/office.lcsoftware.co.za/fullchain.pem
ssl_key = </etc/letsencrypt/live/office.lcsoftware.co.za/privkey.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = </usr/share/dovecot/dh.pem
ssl_prefer_server_ciphers = yes
ssl_min_protocol = TLSv1.2
when trying to connect using PHP i get
imap_open() Certificate failure for office.lcsoftware.co.za: certificate has expired: /O=Digital Signature Trust Co./CN=DST Root CA X3
Any help will be appreciated.
Thank you
Just signed up to comment on my interesting morning.
We have a site using LetsEncrypt SSL linked to Stripe. Automatic subscription renewals were taken this morning, and the callback to our webhooks failed due to SSL error. Site displayed fine in chrome with correct certificate path.
Running a test through the SSL Labs test showed both valid and invalid (expired) paths - SSL Server Test (Powered by Qualys SSL Labs)
Site hosted on Windows. I exported the expired certificates then deleted from the store, forced a refresh of the SSL certificate and then both the SSL labs test and the Stripe callback events passed successfully. Not sure which bit was the fix, but all good now.
Good luck all
1 Like
[seems to be wreaking havoc on really old Windows systems and even to many iOS devices]
You can say that again. I have a bunch of customers with trouble on 'older' machines, with windows 7 & chrome that cannot reach us. Can't reproduce it anywhere except XP on Browserstack (not sure if that is up-to-date), but the errors do not lie: I am seeing people that cannot connect from windows 7 that has issues too.
I'm force renewing everything with the new root only; hopefully that helps 
If not, I'm going to be working a long way into the night figuring out an alternative solution...
1 Like
If anyone is using Traefik with Letsencrypt. Please use the preferred chain ISRG Root X1 in your ACME configuration.
Otherwise, it default fetches the cross signed from DST Root CA X3. So you get the chain mycert > R3 > ISRG Root X1 > DST Root CA X3. And OpenSSL seems to validate the entire chain. Even though ISRG Root X1 is a trusted root CA in your computer. I think LE assumed the cross-signed cert strategy would work in theory. But, in practice OpenSSL doesn't support this.
2 Likes
@Holstvoogd
You may still need to install the "ISRG Root X1" cert in those Win7 PCs.
[depending on how outdated they are]
Self-signed: der, pem
1 Like
I'll have to admit my knowledge about this is bad.
I'll write this response as a help for anyone else facing the same problem.
My setup is like this:
Debian 10 Server -> Docker -> ASP.net Core container running hosting with Kestrel server
I use this software to renew the cert:
It gives me a .crt file with 3 certificates:
- The actual cert for my domain
- R3 (ISRG Root X1) 2020-09-04 -> 2025-09-15
- ISRG Root X1 (2015->2035)
I combine this file + the private key into a PFX file, which I then import and set up SSL with in .NET.
So when this issue arised, I had the expired DST_Root_CA_X3.crt in /etc/ssl/certs on the Debian host and it was also "inside" the Docker container.
For some reason, some mechanism in my setup (Docker, .NET , Kestrel?) seemed to use the expired /etc/ssl/certs/DST_Root_CA_X3.crt and not the "chain" from my .crt file..
After removing this /etc/ssl/certs/DST_Root_CA_X3.crt, and restarting the Docker container, it instead started "serving" the correct, non expired cert-chain.
As for how/why this actually works, I have no clue 
1 Like
If you don't need to support any older Android devices, you could remove the last cert from file:
/etc/letsencrypt/live/office.lcsoftware.co.za/fullchain.pem
Restart server and things should look better.
2 Likes
To me this seems like a partial replay of the Year 2000 problem, however one difference with the Y2K bug was there was enough public paranoia that corporations spent a lot of effort and money on being prepared. So January 2000 there wasn't much issue, which then cause some out-lash that it was all a waste of time, effort, and money. But there wasn't much of a disturbance in the world. I feel that the LE community did do due diligence on address and informing the powers that be, but the paranoia wasn't high enough to make R3 Intermediate certificate expiring to motivate the web world as a whole to be fully prepared and tested prior to R3's expiration. Just one point of view. 
1 Like
Thank you.
removed it and restarted dovecot and postfix, and it's working now.
What happens when the certificate refreshes
1 Like
We're having the same issue with cert-manager on our Kubernetes clusters. Does anyone know how I can set my cluster issuer to use the ISRG Root X1 chain?
Our issuer definition:
Spec:
Acme:
Email: ***
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt
Server: https://acme-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: nginx
Status:
Acme:
Last Registered Email: ***
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/208102080
Conditions:
Last Transition Time: 2021-09-20T16:30:47Z
Message: The ACME account was registered with the ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
According to this page it should just be as simple as setting preferredChain on acme. Could someone please confirm?
preferredChain: "ISRG Root X1"
And are there future implications of this?
You get a new certificate.
You need to restart your services to use that new certificate.
The sun will come up and the stars will shine at night.
Life will go on happily.
[until the next problem shows up]
LOL
2 Likes