Problem with R3 since 15:00 EST. All email down on any iOS device running current iOS 14.x

Since 15:00 today, all email access for any iOS device is broken to our email server using letsencrypt SSL certificate. The certificate path returned by iOS only has our email certificate signed by the expired R3 issued by the old DSL root certificate, not the new R3 issued by the ISRG Root X1 root certificate. If I check the certificate path for the same certificate using Windows 10 or a non iOS device the correct certificate path is returned. There is nothing that can be done on our end. We stopped using the letsencrypt certificate and had to resort to a ZeroSSL certificate (that does not have that problem). This is causing other issue because the certificate is not a wildcard certificate.

The certificate path returned for these tests are returned by the error dialog box or the TLS Inspector app on iOS. On Windows 10 it is returned from clicking on the lock icon on chrome's address bar.

Can you work with Apple to fix the problem? Or can you change the certification path on the certificate that you issue so that it does not use a certificate called R3. Maybe you want to call it R4?

Thanks,
Mario Jauvin

1 Like

This can't be verified by anyone here unless you provide a domain test against. Try getting clients to restart their devices as they may be caching an invalid chain.

2 Likes

Is this a Windows IIS based mail server by chance? Exchange? If so, this worked for me...

Make sure the expired root cert isn't hiding in your Local System cert store.

After a lot of head banging I finally found the one place no one had looked before. The user certificate store for the Local System account. That’s right - there is such a thing and it is not the same as the computer store.

To get to it, you need to download PsTools from SysInternals 363 and run psexec -i -s mmc.exe, go to File -> Add-Remove Snap-in, choose Certificates and My user account. Now look over the cert store for anything R3 expiring in september.

After this you need to “touch” the bindings in IIS (for example, change the certificate and then back again or delete/add the binding) and after IIS is then restarted it will finally start to serve the correct chain.

Ps… the part about “re-creating all HTTPS bindings” was a requirement… Although I didn’t have to delete them entirely. After the expired R3 cert was deleted using psexec, I had restarted IIS and still got the chain error. It wasn’t until I went into the website in IIS and flipped the certificate to something bogus, saved settings, then went back and flipped it again to my LE certificate. Restarted IIS and it FINALLY started serving the correct R3 cert.

3 Likes

Or you can see Let's Encrypt DST Root CA X3 expiry Sept 30th 2021 | Certify The Web Docs and just install Certify The Web (which will perform some basic cert store maintenance including the Local System store) then reboot.

2 Likes

Thanks for the tip. I didn’t know that tool existed. So it babysits the Windows root store and automatically clears invalid certs related to Let’s Encrypt chains?

So I was doing this manually, looks like this tool would bypass the psexec hoop jumping described in my above host.

However in the case of Windows and IIS it didn’t even matter if the root store was configured correctly. Until you actually manually changed the IIS binding to a non-LE cert and back, it was still serving the invalid root. So dumb. I’ve seen this in Windows Server 2016 and 2012 now. Apache much better. :wink:

3 Likes

If you're on IIS 8.5+ (Windows Server 2012 R2 or later), you can use a piece of the Certificate Rebind feature instead of mucking directly with the certificate bindings in order to prod IIS into re-evaluating the chain it's serving without a reboot.

Just run the following using the thumbprint of the cert bound to your site for both the oldcert and newcert arguments:

%SystemRoot%\System32\inetsrv\appcmd.exe renew binding /oldcert:THUMBPRINT /newcert:THUMBPRINT

You can also deal with the cert store issues (including the System user's stores) with a carefully crafted reg file in a pinch rather than dealing with external utilities or software.

3 Likes

Thanks YOU a lot !!!!!!!!!
You save my day!

1 Like

Hi, the devices were rebooted several times and 1 was upgraded from 14.7 to 14.8. I do not have a site for you to test because I removed the letsencrypt certificate since it was not working. The certificate is served by pfSense, a freebsd based firewall router. If I have the oppportunity I will try to fireup a box with the letsencrypt cert.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.