It is spurious, because it is signed by a root CA that has expired, and therefore the signature is invalid. Put another way: if they removed that certificate, the chain would be equally valid.
Most of the articles which described the potential failures on Sep 30 were considering devices which hadn't yet had ISRG Root X1 added to their trust store. They didn't consider devices which already had ISRG X1, but which would still be borked by the expiry of DST X3. This is what has happened here (as experienced by myself and the OP).
The fact that certain API consumers are also affected is more insidious. In any case, the solution is to remove the final certificate from the chain, i.e. the one signed by DST X3 which has expired.
The OP will be able to demonstrate this for themselves. It's technically a client problem: their version of curl/openssl is not performing the correct logic for validating the cert chain (even though ISRG X1 is in their trust store). However, removal of the final cert on the server side will fix this problem. The only negative impact will be that certain really old Android devices (we're talking Android 5 or earlier) will no longer be able to validate the cert.
It's for the benefit of really old Android clients. Many Android devices don't receive updates from their manufacturer any more, and hence don't get updates to their trust store. DST X3 is kept in the chain to keep them happy, and it only works because of a bug in the same old versions of Android which ignores the notAfter date of the root cert.
The certificate which signs ISRG X1 by DST X3 is valid until 2024, even though the DST X3 root itself was only valid to Sep 30 2021. This itself is unusual, but it's enough to keep those Android devices happy.
If he removed the last cert from that one site.
IT WORKS!
[total false sense of accomplishment]
But then every other site that has that long chain continues to fail.
Thus the "FIX" is not really a "FIX" for this at all.
For his system to work, every instance of that cert would have to be removed from the entire Internet.
From his perspective, he can't make changes to all the servers on the Internet.
Do you even know what the difference between a server and a client is?
The problem he sees is fundamentally a client-side problem: i.e. it's his "curl" / "openssl" which is using the wrong rules for certificate validation, and it should ignore the dead certificate. But he can fix it server-side by removing the dead cert from the chain.
That's as long as he doesn't mind breaking ancient Android clients, which have their own different problem.
Still short of that 100k posts read.
Go figure - none of them ever explained to me what a client nor what a server was!
I guess I just never asked the right questions...
If he removed the last cert from that one site.
IT WORKS!
[total false sense of accomplishment]
But then every other site that has that long chain continues to fail.
Thus the "FIX" is not really a "FIX" for this at all.
So, it's a question of the role you take in this.
If you are a server administrator, and you're getting complaints from your clients that they can't connect to your system, then either you make a single change to your server and they are happy, or you tell every single client how to fix their system (e.g. by removing the expired DST X3 from their trust store). I note that nobody on this thread has yet suggested how the OP do this.
If you are a user, who finds "oops - a whole bunch of sites seem to be broken now", then you fix your client.
It's also a question of power relationships. In our case, we had a large company (Zendesk) trying to talk to the API on our self-hosted Jira server, which we sign with LetsEncrypt. We opened a case with Zendesk when it failed. They were unable to fix it. So in the end, we removed the dead cert from the chain, and hey presto, Zendesk started working again. They're the big company, we're the minnow, we can't force big companies to have a clue. So pragmatically, we fix the problem and move on.
From my point of view, it's unfortunate that LetsEncrypt prioritises one set of obsolete clients, over a different set of slightly less obsolete (and to me, more important) clients. I think they should have dropped the DST X3 cert once the DST X3 root expired.
I've even spun up a similar test system, just to see if it is broken:
cat /etc/issue
Ubuntu 20.04.3 LTS \n \l
But it works just fine.
The problem discussed here was clearly created by some action taken on that system.
Possibly an install or update that went awry or stomped on something it should not have...
I would recommend (get your data out and) flattening it and starting over.
This time make sure curl continues to work after each change.
[Sorry for being away for so long, but I had to be at my desktop and of clear mind to follow what is an interesting thread.]
the above test returns:
curl: option -CAcert: expected a proper numerical parameter
yes the curl build is for an older Apple system (sorry, but Apple's OS churn gives old a new meaning). However it is occuring on other clients (I will get the system references)
no, connection problems are not apparent; responses are always occuring.
So, Let's Encrypt made a decision to not drop DST X3 cert; it would be proper of them to illustrate why... en passant. as this is a bigger problem.
The questions now become:
a) '...and starting over. [...] make sure curl continues to work after each change.' Spinning up a new server is perfectly doable. This is an experiment that does require some time to afford it proper attention and will be done in due course.
b) The need for security is on the server side (updates). The clients are connecting with tokens that are not even being communicated to them via the web, so the server is running securely. The clients can run curl with the -k flag and still verify via https that cert validity. What is weak in this scenario, temporary as it may be?
"For "b": however they would have to posses the token (which is not 2 characters long) to successfully be accepted by the application. Is my risk assessment wrong?
Unless the token is being used to encrypt the connection (within the https connection), yes. your assumption is wrong.
Call me (secure call)
I'll call your bank (secure call)
Then anything they ask me, I ask you
You tell me and I tell your bank
Soon I will be into your bank as you.
