Connection Errors on Apple Devices

The reason seems to be that mac OpenSSL is built with LibreSSL, which cares about the expired-root in ways that upstream OpenSSL doesn’t.

3 Likes

Interesting. Searching quickly on Google I found this problem open on GitHub from LibreSSL. Maybe there's a connection? The scenario reported in the problem is the same, unable to validate an alternative chain after the expiration of the root certificate.

3 Likes

I wonder if Apple would be willing to ship an OS update that changes this behavior. It would be great for site operators not to have to choose between losing compatibility with (some versions of) macOS and losing compatibility with (some versions of) Android.

1 Like

I wonder how intentional this divergence from OpenSSL is. There’s an argument to make that the verification logic should verify the very chain I give it, and nothing else.

It was news to me today that it’s apparently OK for TLS clients to validate against an expired root. If that’s the case then I wonder why roots have expiry dates at all.

2 Likes

I think it's been a contentious issue. Here's a take from Ryan Sleevi from last year:

Ryan emphasizes that there isn't just one chain that justifies trusting a certificate, but there can be several (and perhaps different clients will disagree on the specific justifications that they accept).

3 Likes

To let you know when to remove them from the non-expiring zone - LOL

The multiple-paths thing is familiar enough, but it was news to me today that a trust anchor is legitimate without an expiry date, even if its expression as a certificate contains a notAfter date (that isn’t freakishly far into the future, that is).

Very unlikely. They don't even provide security updates for older hardware that can't run their newest OS.

Site operators have another choice: don't use Let's Encrypt. :slight_smile:

Thanks for this! I run some buildbots deliberately using older macOS and their git and svn have been unable to pull new code since yesterday.

What worked for me on macOS 10.10 and 10.11 was:

  • with Safari, download old Firefox 78 ESR: https://download.mozilla.org/?product=firefox-esr-latest-ssl&os=osx&lang=en-US (newer Firefox don't support old macOS)
  • with Firefox, download this file: https://letsencrypt.org/certs/isrgrootx1.der (Safari can't download it because letsencrypt.org itself serves the problematic expired cert.)
  • drag the downloaded file to KeyChain Access.app
  • when prompted, choose to add it to the "System" keychain (you'll need admin password)
  • in Keychains > System, find "ISRG Root X1", it should have a red X on its icon
  • double click it, a new window appears
  • under the Trust section, for "when using this certificate" choose "Always Trust"
  • close window, enter your admin password again

After that my svn up and git pull now work again from various servers.

4 Likes

So adding "ISRG Root X1" fixed macOS 10.10 and 10.11, but macOS 10.12 and later already include "ISRG Root X1" by default, yet some versions are giving me trouble, others not. I think it may depend on OpenSSL/LibreSSL versions. I made a little table:

macOS   OpenSSL version
10.10.5 OpenSSL 0.9.8zh
10.11.6 OpenSSL 0.9.8zh
10.12.6 OpenSSL 0.9.8zh
10.13.6 LibreSSL 2.2.7
10.14.6 LibreSSL 2.6.5
10.15.7 LibreSSL 2.8.3
11.6    LibreSSL 2.8.3
12.0    LibreSSL 2.8.3

The ones with older LibreSSL are giving me trouble, and I think that may be why...

3 Likes

FYI - more information on LibreSSL. The latest stable release is 3.3.4

I can Confirm a fix that worked on both a 10.9.5 & 10.11.6 Mac OS users. Simply set the DST Root CA X3 to "Always Trust" on several Mac's I manage in an office and home's this fix work for 4 websites that previously had issues with this CERT ERR.

https://www.nynewspapers.com/

Directions for fix:

  1. Open ~/Applications/Utilities/Keychain Access.app
  2. From View menu select "Show Expired Certificates"
  3. On the Left Sidebar pick System Root
  4. In search bar top-right type DST
  5. Double-click "DST Root CA X3"
  6. In pop-up, turn down "Trust" arrow and set "When using this certificate" to "Always Trust"
  7. Close the pop-up and put in an Administrator user/password info.
  8. Close all open Browsers & Keychain you should be good to go after that.

Yeah, but even the still-beta macOS 12.0 Monterey uses the 3 year old 2.8.3!

2 Likes

Kinda sad. :slightly_frowning_face:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G9323

$ /usr/bin/curl -I https://www.letsencrypt.org
curl: (60) SSL certificate problem: certificate has expired
...

$ /usr/bin/curl -I https://www.steampowered.com
curl: (60) SSL certificate problem: certificate has expired
...

$ /usr/bin/curl -I https://opendns.com
curl: (60) SSL certificate problem: certificate has expired
...

We really need the Internet to work again, and we need a solution that that does not require intervention on the part of every user, or every server administrator, or every OS vendor. Fix Let's Encrypt so that it works again.

Sadly, such a "simple solution" doesn't exist - or it would have been applied already.
The biggest part of the Mac OS X not being able to correctly identify those certs as valid (not expired) is within the control of Apple - not Let's Encrypt.
That said, some clever humans (outside of the Apple corporateshpere) have been able to find manual was to "fix" the problem. But most of those fixes will require "intervention" (until an official update from Apple is released - I don't Apple, so I don't know if one is out already or even in the works for your specific O/S version) on those affected client devices.

2 Likes

It is completely inconceivable that it is not within Let's Encrypt's power to make their own web site and their service continue to function on a major OS version released only three years ago and which did receive a security update just a couple months ago.

It's also inconceivable that Let's Encrypt did not test the effects of the root certificate transition on their web site or service with such an OS version ahead of time to develop a solution that would have made for a smooth transition, or that Let's Encrypt performed such testing and deemed the extensive breakage this transition has caused to be acceptable.

It is not about "my" OS version. I am a software developer supporting users who use a wide range of macOS versions. Not all users are able to update to the latest version of the OS.

1 Like

That's probably all very conceivable I think, but probably not constructive to complain about at this moment. I think working on a solution is more constructive than the blaming part. I'm sure Let's Encrypt will review and reconstruct what happened the last few days and learn from it. I hope.

I don't recall asking anyone to update anything.
But since you didn't really address the post to anyone specifically, I'm not sure to whom you are speaking with.
But I'll play along anyway...
What would you have anyone do now (now that the the cert that has been expiring for 20 years has finally expired) at this time?
Going forward, is there a simple solution?
Didn't the expiry warnings go on deaf ears? [don't shoot the messengers]

As for your first paragraph:
You can please some of the people all of the time.
All of the people some of the time.
But you really can't please all of the people all of the time.
And this is one of those moments when this new cert just can't please all of the people.

As for the second paragraph:
Do you think anyone has the resources to test every combination of system that is out there in a situation that hasn't happened (to this scale) before?
Devices are "living" way past their supported lifespans.
Many "current" devices aren't really running software that is actually that "current".
This is a not-for-profit organization that is doing far more than is literally can afford to do already.
Most of the people you "speak" with here (myself included) are merely longtime volunteers.
This is big corporate business trying to pull a fast one and get anyone to reach into their wallets or else.
It is unfortunate.
It is a huge learning experience (for most of everyone on the Internet).
It is an opportunity.
It is a realization that what we all thought it was safe and secure was actually very unsafe and insecure.
[the curtains have been drawn - they light has been shone on many dark corners]
It was not intentional.
It would have happened (if not this way, in some other way).

But I'm biased - my thoughts are skewed - my perspective limited.
Thus this dialog.
Do you see me?
I'm trying to see you...

1 Like

I had success (more or less) on 10.6.8. The existing Firefox wouldn't download isrgrootx1.der exactly, but it did prompt me to trust it, so firefox works again. Yay! Then I downloaded it with wget (from macports which already worked), and dragged it to KeyChain Access.app which only let my user trust it in the "local" keychain.

I tried to add it to the "System Roots" keychain but it didn't work (but there were no errors when I tried, it just didn't end up in the list). Then I followed instructions at this link to export the root certificates from a later macOS and import them:

safari - How do I update my root certificates on an older version of Mac OS (e.g. El Capitan)? - Ask Different

The new certificates ended up in the "System" keychain (not "System Roots"). It was probably unnecessary, but I changed the script in the above link so as to add them to the "System Roots" keychain, and imported the new roots there. They were imported successfully.

However, Safari still doesn't work, but I'm pretty sure that that's for other reasons. I think it only supports TLSv1.0 and letsencrypt.org and my sites don't support that. Firefox and macports tools all work, but macports already did anyway. That's the main thing.

But I suspect that any other ancient Appleware on that host won't work anymore even with new root certificates, but that's probably to be expected. Luckily, I don't use that host for its fancy webbed footwork.

1 Like