Help thread for DST Root CA X3 expiration (September 2021)

Thank you so much everyone.

Using snap to run a newer version of certbot, and telling it to use --preferred-chain "ISRG Root X1" worked like a charm. Now my phone is back to happily running DOT through Stunnel.

The preferred chain option was automatically added to /etc/letsencrypt/renewal/MyServer.conf as:
preferred_chain = ISRG Root X1

So on next renewal it will not break the chain.

2 Likes

So then what did you do after getting rid of ca x3, recreate the certificate?

Wayne Sallee

1 Like

I got:
update-ca-certificates
/usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt not found, but listed in /etc/ca-certificates.conf.

So in /etc/ca-certificates.conf I changed
mozilla/DST_Root_CA_X3.crt
to
!mozilla/DST_Root_CA_X3.crt

Then re-ran
update-ca-certificates

The fact that my server is still running debian 9 is playing into the problem. Time to upgrade.

Wayne Sallee

2 Likes

My /etc/ca-certificates.conf is dated 25th September and the mozilla/DST_Root_CA_X3.crt entry already was commented out with a '!'.

However, it is also listed in /snap/core18/2128/etc/ca-certificates.conf and I've already found that trying to change that results in it being marked 'broken'.

1 Like

If it can help anyone, the problem of wrong chain response for asp.net application in my case resolved deleting certificates cache, on linux machine /home/username/.dotnet/corefx/cryptography/x509stores/ca
and restarting application

3 Likes

A post was split to a new topic: Nginx with preferred-chain "ISRG Root X1" - still showing "DST Root CA X3"

A post was split to a new topic: Errors using FPDF to make PDFs from websites

A post was split to a new topic: Alpine Linux issue

A post was split to a new topic: Splitting Wishes

wouldn't we have the same problem at some point in the future if we switched to another CA? or was there a problem the way letsencrypt implemented this?

getting pressure from my higher-ups to make our site work for older clients and it sounds like the only way is to switch CAs?

1 Like

Yes, this can happen to any CA that experiences a root expiry.
Switching to a newer root implies that clients made/unmodified from before the replacement root was issued would have no knowledge of that second root (only the first one - which has then expired).

But of course this example/lesson will serve plenty to all that may come after.
The first time it is a mistake.. unavoidable/unknown situation.
The second time - lord help them!

Older clients should be able to work with the default chain (through "DST Root CA X3").
But if you had to switch off that (to "ISRG Root X1") because newer clients had problems...
Then you can't really satisfy them both with the current LE chains.
I'd switch to an alternate (free) CA that supports ACME protocol. Like: ZeroSSL.com or BuyPass.com

2 Likes

Inevitably yes, probably. In an ideal situation, you have multiple certificates in all the root stores, old and new, overlapping each other with much overlap.

However, Let's Encrypt is relatively new. The ISRG Root X1 is already in all major root stores for more than 5 years. (!!!) Unfortunately, not every piece of software has been updated.. And LE didn't get a new cross-sign: it was time to stand on their own.

Now, you can blame Let's Encrypt for that. But IMO that's not fair. And it's probably also a matter of taste. But in my opinion if your software is SO old it doesn't work properly at the moment, that device shouldn't be on the internet anyway.

3 Likes

yes totally agree the client should be updated or taken offline. problem is if my competitors allow traffic from these older clients and we do not then we lose customers. trying to find a way in haproxy to identify these clients and redirect them to a webpage but not having much luck.

1 Like

Inconvenient but a pre-connection page with two continue buttons might do the trick.
One button served from an FQDN that has cert trust path #1
Other button served from another FQDN that has cert trust path #2
If they can see the button, they can reach the site (that serves it).

Hi sorry for the construction inconvenience, but we are working on the site and you only need to enter it via the door that is open/working for you today.
[button one] [button two]
Again, sorry for all the dust and noise too...

:jackhammer:
:drill:
:wood:
:sheetrock:
LOL

1 Like

Well! I have very serious issues with connecting from iOS devices (ver 14 and 15) to adobe connect servers. I have tried many ways and yet no sucess. As we we are providing online education medium, an urgent answer is a huge help.
Looking forward to hearing from you.

1 Like

Hi @sherrikalak,

ISRG Root X1 has been supported since iOS 10 and so in principle these devices should still be able to interoperate with sites using Let's Encrypt certificates. An error in connecting would probably mean that the intermediate certificate sent by the server is incorrect. (For example, it could have hardcoded an expired intermediate certificate, or fail to send any intermediate certificate at all.)

Could you give an example of a domain name that you're having problems connecting to? Alternatively, you could test with https://ssllabs.com/ and look for any errors in the "Chain issues" section, which would likely provide a useful diagnosis.

3 Likes

Hi @sherrikalak welcome to the LE community forum :slight_smile:

That is key to understanding the problem.

1 Like

A bit off topic but...

It's unfortunate that IdenTrust didn't consider the issues they would cause when they issued their root certificate on September 30, 2000.

That aside, this is a good thing for web PKI. Many root certificates were issued in the early 2000s, this type of expiry is going to happen much more frequently.

Because of this unavoidable event, Let's Encrypt is shaking things up a bit and seeing what bolts fall out. We can learn from this event so future root expiration are handled more gracefully.

EDIT: I do not appreciate my posts being edited without my knowledge. Please see the original quoted context here Help thread for DST Root CA X3 expiration (September 2021) - #1208 by datadragon61

3 Likes

Thanks Schoen! Your tip about ssllabs.com saved me! I directed the admins to repair the chain and it seems it worked!

5 Likes

The reissue problem with acme.sh drove me nuts.

According to https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain the default is still 'DST Root CA X3'.

But even when I force reissued with --preferred-chain "ISRG Root X1" or Le_PreferredChain='ISRG Root X1' I got the fullchain with the now expired 'DST Root CA X3' in the chain

Certificate chain
 0 s:/CN=domain.tld
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Somebody said that they had to re-issue the certificate. Since force didn't work. I tried to delete and issue again only to learn that acme.sh now uses zerossl by default.
https://community.letsencrypt.org/t/the-acme-sh-will-change-default-ca-to-zerossl-on-august-1st-2021/144052

Came across the switch to continue using letsencrypt

acme.sh --issue --domain 'domain.tld' --dns dns_pdns --debug --server letsencrypt

Trying to switch back with this is still unconfirmed since I had to give up because of the rate limits.

I really would have appreciated LE raising those limits during such vital changes.

For now I had to switch to zerossl to keep everything running

And I still don't know if it will be fine next time since I can only continue my tests next week.

1 Like