Yes. Ignoring the certificate on the curl call with -k processes the action as expected.
But this doesn't add up:
How can the chain validation in your curl be "broken"/"failing" ?
1 Like
Please do:
sudo apt-get update
sudo apt update
Then show the output of:
sudo apt list --upgradable
1 Like
fwupd-signed/focal-updates 1.27.1ubuntu5+1.5.11-0ubuntu1~20.04.2 amd64 [upgradable from: 1.27.1ubuntu2+1.3.11-1~focal1]
fwupd/focal-updates 1.5.11-0ubuntu1~20.04.2 amd64 [upgradable from: 1.3.11-1~focal1]
libfwupd2/focal-updates 1.5.11-0ubuntu1~20.04.2 amd64 [upgradable from: 1.3.11-1~focal1]
libfwupdplugin1/focal-updates 1.5.11-0ubuntu1~20.04.2 amd64 [upgradable from: 1.3.11-1~focal1]
Hmm. quantity = 4. Seen before...
Nothing stands out there...
Please show:
apt list --installed | grep curl
I get:
curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
libcurl3-gnutls/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
Also, what else have you installed on this server?
1 Like
curl/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
libcurl3-gnutls/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
libcurl4-openssl-dev/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
libcurl4/focal-updates,focal-security,now 7.68.0-1ubuntu2.7 amd64 [installed]
I usually setup with the following packages:
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev software-properties-common libffi-dev dirmngr gnupg apt-transport-https ca-certificates redis-server redis-tools nodejs yarn
sudo apt-get install libgdbm-dev libncurses5-dev automake libtool bison libffi-dev
sudo apt-get install imagemagick
sudo apt-get install mosh
I do not believe there are any other packages as this server has been spinned up not even 3 months ago
I also see exactly this issue with macOS 10.14.6, using either curl or openssl, and I can explain what's going on.
$ curl https://tuttopepe.saltalafila.online
curl: (60) SSL certificate problem: certificate has expired
...
$ openssl s_client -connect tuttopepe.saltalafila.online:443 -servername tuttopepe.saltalafila.online -showcerts
CONNECTED(00000005)
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
...
In short, the solution is to remove the final cert from the chain (subject ISRG X1, issuer DST X3) on the server, from the chain.pem or fullchain.pem.
Longer version: LetsEncrypt now gives a chain that looks like this:
<your server signed by LetsEncrypt R3>
|
<LetsEncrypt R3 signed by ISRG X1>
|
<ISRG X1 signed by DST X3>
The final cert was included because in early days, many systems didn't have ISRG X1 in their trust store, but they did have DST X3.
These days, the chain of trust should end on the 2nd certificate (since the ISRG X1 root is now in the trust store), and the 3rd one should be ignored. But actually what happens with some versions of openssl is that it continues the chain to the third, finds DST X3 in the trust store has expired, and then treats the whole chain as expired. Oops. The expiry date Sep 30 14:01:15 2021 GMT is the expiry date of the DST X3 root certificate itself - Google for this date to turn up more info such as
The question then is, why does LetsEncrypt still include the last cert in the chain, if it's useless? Apparently it's for the benefit of certain old Android devices, which (a) don't have the ISRG X1 in their trust store, and (b) have a convenient bug which ignores the expiry date on root certificates in their trust store.
Unfortunately, this approach is borking a number of other clients as well, as you've found.
I use "dehydrated" to issue certs, and I can work around this problem by adding
PREFERRED_CHAIN="ISRG Root X1"
to the config file; it then trims off the spurious cert from the end. I don't know how you'd do this with other clients.
It's not quite as big a problem as you might think though. Whilst "curl" and "openssl" on my mac demonstrate this problem, browsers don't. It seems they all have their own TLS stacks built in these days.
However I have had to debug this problem on a number of API-based systems: for example we had a problem where Zendesk would refuse to talk to our Jira server's API after 30 Sep. It was exactly this problem. Removing the spurious certificate from the end of the chain fixed it.
It sounds like you might not have read anything about that cert.
"DST Root CA X3" is the root at the end of the chain on this very site.
See:
Chain of Trust - Let's Encrypt (letsencrypt.org)
echo | openssl s_client -connect community.letsencrypt.org:443 -servername communityletsencrypt.org | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.discourse.org
verify return:1
DONE
CONNECTED(00000005)
---
Certificate chain
0 s:CN = *.discourse.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
2 Likes
[going incommunicado for a number of hours now. Just as the plot might reveal itself...]
1 Like
It is spurious, because it is signed by a root CA that has expired, and therefore the signature is invalid. Put another way: if they removed that certificate, the chain would be equally valid.
Most of the articles which described the potential failures on Sep 30 were considering devices which hadn't yet had ISRG Root X1 added to their trust store. They didn't consider devices which already had ISRG X1, but which would still be borked by the expiry of DST X3. This is what has happened here (as experienced by myself and the OP).
The fact that certain API consumers are also affected is more insidious. In any case, the solution is to remove the final certificate from the chain, i.e. the one signed by DST X3 which has expired.
The OP will be able to demonstrate this for themselves. It's technically a client problem: their version of curl/openssl is not performing the correct logic for validating the cert chain (even though ISRG X1 is in their trust store). However, removal of the final cert on the server side will fix this problem. The only negative impact will be that certain really old Android devices (we're talking Android 5 or earlier) will no longer be able to validate the cert.
Although it is expired, I stand by my statement:
Do you know why it is there?
I mean, if it is obviously expired, why is it still being used?
Here is some reading material:
DST Root CA X3 Expiration (September 2021) - Let's Encrypt (letsencrypt.org)
And a video too:
3 Likes
Incidentally, the problem that the DST X3 cert gives with OpenSSL 1.0.2 is described in more detail here, together with several workarounds:
https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
What I've described is what they call "workaround 3", where you make a change in one place (on the server).
The other workarounds have to be applied on every affected client.
From his perspective, he can't make changes to all the servers on the Internet.
It is Ubuntu 20.
It should NOT fail with either chain.
2 Likes
Yes I do. Please don't patronise me.
It's for the benefit of really old Android clients. Many Android devices don't receive updates from their manufacturer any more, and hence don't get updates to their trust store. DST X3 is kept in the chain to keep them happy, and it only works because of a bug in the same old versions of Android which ignores the notAfter date of the root cert.
The certificate which signs ISRG X1 by DST X3 is valid until 2024, even though the DST X3 root itself was only valid to Sep 30 2021. This itself is unusual, but it's enough to keep those Android devices happy.
Well you do sound uninformed.
OR informed just enough to be dangerous - LOL
Here is a whole lot on this topic:
Production Chain Changes - API Announcements - Let's Encrypt Community Support (letsencrypt.org)
If he removed the last cert from that one site.
IT WORKS!
[total false sense of accomplishment]
But then every other site that has that long chain continues to fail.
Thus the "FIX" is not really a "FIX" for this at all.
For his system to work, every instance of that cert would have to be removed from the entire Internet.
1 Like
From his perspective, he can't make changes to all the servers on the Internet.
Do you even know what the difference between a server and a client is?
The problem he sees is fundamentally a client-side problem: i.e. it's his "curl" / "openssl" which is using the wrong rules for certificate validation, and it should ignore the dead certificate. But he can fix it server-side by removing the dead cert from the chain.
That's as long as he doesn't mind breaking ancient Android clients, which have their own different problem.
hmm...
They each have six letters...
No do tell me!
OK GOT IT!
Next time I get a headache... I'll tell my wife to take two aspirins!
Just to recap: There is only one problem here (not two).
2 Likes
I guess I haven't been around the block as many times as you nor read as much:
Still short of that 100k posts read.
Go figure - none of them ever explained to me what a client nor what a server was!
I guess I just never asked the right questions...
3 Likes
If he removed the last cert from that one site.
IT WORKS!
[total false sense of accomplishment]
But then every other site that has that long chain continues to fail.
Thus the "FIX" is not really a "FIX" for this at all.
So, it's a question of the role you take in this.
If you are a server administrator, and you're getting complaints from your clients that they can't connect to your system, then either you make a single change to your server and they are happy, or you tell every single client how to fix their system (e.g. by removing the expired DST X3 from their trust store). I note that nobody on this thread has yet suggested how the OP do this.
If you are a user, who finds "oops - a whole bunch of sites seem to be broken now", then you fix your client.
It's also a question of power relationships. In our case, we had a large company (Zendesk) trying to talk to the API on our self-hosted Jira server, which we sign with LetsEncrypt. We opened a case with Zendesk when it failed. They were unable to fix it. So in the end, we removed the dead cert from the chain, and hey presto, Zendesk started working again. They're the big company, we're the minnow, we can't force big companies to have a clue. So pragmatically, we fix the problem and move on.
From my point of view, it's unfortunate that LetsEncrypt prioritises one set of obsolete clients, over a different set of slightly less obsolete (and to me, more important) clients. I think they should have dropped the DST X3 cert once the DST X3 root expired.
I also agree that the other chain should have been the default - let those that need Oldroid compatibilty explicitly do so.
But this topic isn't trying to address both sides.
The client side is broken!
2 Likes