Unclear motive for negative server responses

It is spurious, because it is signed by a root CA that has expired, and therefore the signature is invalid. Put another way: if they removed that certificate, the chain would be equally valid.

Most of the articles which described the potential failures on Sep 30 were considering devices which hadn't yet had ISRG Root X1 added to their trust store. They didn't consider devices which already had ISRG X1, but which would still be borked by the expiry of DST X3. This is what has happened here (as experienced by myself and the OP).

The fact that certain API consumers are also affected is more insidious. In any case, the solution is to remove the final certificate from the chain, i.e. the one signed by DST X3 which has expired.

The OP will be able to demonstrate this for themselves. It's technically a client problem: their version of curl/openssl is not performing the correct logic for validating the cert chain (even though ISRG X1 is in their trust store). However, removal of the final cert on the server side will fix this problem. The only negative impact will be that certain really old Android devices (we're talking Android 5 or earlier) will no longer be able to validate the cert.

Although it is expired, I stand by my statement:

Do you know why it is there?
I mean, if it is obviously expired, why is it still being used?

Here is some reading material:
DST Root CA X3 Expiration (September 2021) - Let's Encrypt (letsencrypt.org)
And a video too:


Incidentally, the problem that the DST X3 cert gives with OpenSSL 1.0.2 is described in more detail here, together with several workarounds:

What I've described is what they call "workaround 3", where you make a change in one place (on the server).

The other workarounds have to be applied on every affected client.

From his perspective, he can't make changes to all the servers on the Internet.
It is Ubuntu 20.
It should NOT fail with either chain.


Yes I do. Please don't patronise me.

It's for the benefit of really old Android clients. Many Android devices don't receive updates from their manufacturer any more, and hence don't get updates to their trust store. DST X3 is kept in the chain to keep them happy, and it only works because of a bug in the same old versions of Android which ignores the notAfter date of the root cert.

The certificate which signs ISRG X1 by DST X3 is valid until 2024, even though the DST X3 root itself was only valid to Sep 30 2021. This itself is unusual, but it's enough to keep those Android devices happy.

Well you do sound uninformed.
OR informed just enough to be dangerous - LOL
Here is a whole lot on this topic:
Production Chain Changes - API Announcements - Let's Encrypt Community Support (letsencrypt.org)

If he removed the last cert from that one site.
[total false sense of accomplishment]
But then every other site that has that long chain continues to fail.
Thus the "FIX" is not really a "FIX" for this at all.
For his system to work, every instance of that cert would have to be removed from the entire Internet.

1 Like

From his perspective, he can't make changes to all the servers on the Internet.

Do you even know what the difference between a server and a client is?

The problem he sees is fundamentally a client-side problem: i.e. it's his "curl" / "openssl" which is using the wrong rules for certificate validation, and it should ignore the dead certificate. But he can fix it server-side by removing the dead cert from the chain.

That's as long as he doesn't mind breaking ancient Android clients, which have their own different problem.

They each have six letters...
No do tell me!

Next time I get a headache... I'll tell my wife to take two aspirins!

Just to recap: There is only one problem here (not two).


I guess I haven't been around the block as many times as you nor read as much:

Still short of that 100k posts read.
Go figure - none of them ever explained to me what a client nor what a server was!
I guess I just never asked the right questions...


If he removed the last cert from that one site.
[total false sense of accomplishment]
But then every other site that has that long chain continues to fail.
Thus the "FIX" is not really a "FIX" for this at all.

So, it's a question of the role you take in this.

If you are a server administrator, and you're getting complaints from your clients that they can't connect to your system, then either you make a single change to your server and they are happy, or you tell every single client how to fix their system (e.g. by removing the expired DST X3 from their trust store). I note that nobody on this thread has yet suggested how the OP do this.

If you are a user, who finds "oops - a whole bunch of sites seem to be broken now", then you fix your client.

It's also a question of power relationships. In our case, we had a large company (Zendesk) trying to talk to the API on our self-hosted Jira server, which we sign with LetsEncrypt. We opened a case with Zendesk when it failed. They were unable to fix it. So in the end, we removed the dead cert from the chain, and hey presto, Zendesk started working again. They're the big company, we're the minnow, we can't force big companies to have a clue. So pragmatically, we fix the problem and move on.

From my point of view, it's unfortunate that LetsEncrypt prioritises one set of obsolete clients, over a different set of slightly less obsolete (and to me, more important) clients. I think they should have dropped the DST X3 cert once the DST X3 root expired.

I also agree that the other chain should have been the default - let those that need Oldroid compatibilty explicitly do so.

But this topic isn't trying to address both sides.
The client side is broken!


One problem, multiple workarounds.

I was online the day that was posted here.
And I've reposted it dozens of times.
It doesn't apply here.


I've even spun up a similar test system, just to see if it is broken:

cat /etc/issue
Ubuntu 20.04.3 LTS \n \l

But it works just fine.
The problem discussed here was clearly created by some action taken on that system.
Possibly an install or update that went awry or stomped on something it should not have...
I would recommend (get your data out and) flattening it and starting over.
This time make sure curl continues to work after each change.


I see that we have not completely ruled out a .curlrc interfering. Would you try:

curl -CAcert /etc/ssl/certs/ca-certificates.crt https://tuttopepe.saltalafila.online/api/v1/articles/array

It's a easy test and maybe we will get lucky :slight_smile:


This looks like it's a curl build for (old) Apple systems, not Ubuntu. It also seems to be using the OS X TLS stack.

Are you having the connection problems on your Ubuntu server, or on your Apple computer?


[Sorry for being away for so long, but I had to be at my desktop and of clear mind to follow what is an interesting thread.]

  1. the above test returns:
    curl: option -CAcert: expected a proper numerical parameter
  2. yes the curl build is for an older Apple system (sorry, but Apple's OS churn gives old a new meaning). However it is occuring on other clients (I will get the system references)
  3. no, connection problems are not apparent; responses are always occuring.

So, Let's Encrypt made a decision to not drop DST X3 cert; it would be proper of them to illustrate why... en passant. as this is a bigger problem.

The questions now become:

a) '...and starting over. [...] make sure curl continues to work after each change.' Spinning up a new server is perfectly doable. This is an experiment that does require some time to afford it proper attention and will be done in due course.

b) The need for security is on the server side (updates). The clients are connecting with tokens that are not even being communicated to them via the web, so the server is running securely. The clients can run curl with the -k flag and still verify via https that cert validity. What is weak in this scenario, temporary as it may be?

1 Like

This may have way more information that you seek... But it does cover the question you ask.
There is even a link to a video that explains the reasoning.
See: Production Chain Changes - API Announcements - Let's Encrypt Community Support (letsencrypt.org)

I don't really see a question in "a".

For "b": Running curl with -k is not safe at all. It would allow a MITM to use any cert at all and succeed.

1 Like

"For "b": however they would have to posses the token (which is not 2 characters long) to successfully be accepted by the application. Is my risk assessment wrong?

1 Like

Unless the token is being used to encrypt the connection (within the https connection), yes. your assumption is wrong.

Call me (secure call)
I'll call your bank (secure call)
Then anything they ask me, I ask you
You tell me and I tell your bank
Soon I will be into your bank as you.