IIS 8.5 building incorrect chain with Lets Encrypt Authority X3


#90

YES! I can confirm this works.

I used the glorious psexec -i -s mmc.exe, removed the X1 certificate from the intermediate store, did an IISreset, removed the existing bindings and reapplied them. And now it works!

@Knagis you are a great man, thank you!


#91

There is an important catch!

You will have to renew all certs hosted in the IIS from X1 to X3 before applying this solution.

If you don’t do that, your old X1 certs will stop working correctly and their SSL Labs grade will be capped to B.

That’s exactly what happened in ours webservers, which hosts hundreds of LE domain certs.

Am I right?


#92

Yesssssss! Super Knagis!

Problem solved


#93

good way. that is right… :smiley:


#94

@Knagis, thank you for the clear directions! I’m glad folks are finding this fixes their problems. @actyler1001, does this work for you?

@_xentia, @Rouzax, @SanderAtSnakeware, is there something different between @Knagis’ instructions and mine that fixed your problem? I think what @Knagis describes is very similar to what I said, and I want to make sure I capture the distinction so I can explain it well to others. Is the difference that PsUtil allows you to run MMC on a remote machine, and previously you were running MMC on a local machine?

Thanks,
Jacob


#95

The difference is using psexec.exe to launch the management console while impersonating the system account. So that when you choose “My user account” it actually loads not your but the store of the Local System account.


#96

@jsha - your instructions was overlap @Knagis solution at almost 99%. Main difference is that tool - psexec - which allow to run mmc for local user (!). I’m in deep research now - what can be a simpliest way to reach that store - without additional tools - to reach that cert store. Still can’t believe that in Windows server i’ll need a add tool to comply that task


#97

Great job! Thank you very much for sharing! :slight_smile:
I was checking the personal cert store of all user accounts on the server and double and tripple checked cert caches but I wouldn’t come up with the idea to run certmgr with psexec…
:white_check_mark:


#98

SysInternals is a Microsoft Toolset, is it not? :slight_smile:


#99

You are absolutely right @Osiris. My mistake - i’ll correct my post


#100

Thanks for the additional explanation @Knagis, @_xentia. That makes sense.


#101

Your method of using “My user account” with MMC worked for me. I had been using Computer Account and not been able to find the offending cert.


#103

Bravo sir! I just tried this myself and I can confirm it works. Great work.

Ps… the part about “re-creating all HTTPS bindings” was a requirement… Although I didn’t have to delete them entirely. After the old X1 intermediate cert was deleted using psexec, I had restarted IIS and still got the chain error. It wasn’t until I went into the website in IIS and flipped the certificate to something bogus, saved settings, then went back and flipped it again to my LE certificate. Restarted IIS and it FINALLY started serving the X3 intermediate. Wow Windows 2012 R2, wow…


#104

The thing that baffles me is that there is a difference between the local computer and local system account regarding the certificate stores. Very good thinking to try the PSexec tool in order to check this.

@jsha it is indeed the same procedure, but with the local system account thrown in which you can’t access using the regular MMC method.


#105

Thanks for this great find! I also didn’t think that there may be an certificate store for the SYSTEM account. But it actually makes sense since this is the default account for Windows Services and the “WWW Publishing Service” (IIS) runs as System. Also, AFAIK IIS uses the Microsoft HTTP Api to register for HTTP Requests which binds the TCP ports as the SYSTEM process (PID 4) which also uses the system account (but I don’t know the internals).

The difference between “local computer” and “system” certificate storages (as far as I understand) is that the former is visible to all users except for private keys (and is writeable only with administrator rights), while the latter is visible only to the “system” user account (which also happens for other user accounts).

I have used SysInternals ProcessMonitor to check where the intermediate certificates are stored, and it seems they are stored in the registry at <User>\Software\Microsoft\SystemCertificates\CA\Certificates.

The SYSTEM account has a well-known SID of S-1-5-18, so you should find the certificates for the SYSTEM account in HKEY_USERS\S-1-5-18\Software\Microsoft\SystemCertificates\CA\Certificates, the ones for the current user in HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\CA\Certificates and the ones for the Local Computer (which are visible for all users) in HKEY_LOCAL_MACHINE\Software\Microsoft\SystemCertificates\CA\Certificates.

For example, with the following quick&dirty C# program you can list all intermediate certificates for the Local Computer (all users), SYSTEM and the current User account (it will display the registry key name, Subject, Issuer and Thumbprint of the certificate):

using System;
using System.Security.Cryptography.X509Certificates;
using Microsoft.Win32;

namespace ShowIntermediateCertificates
{
    class Program
    {

        private const string userRegKey = @"Software\Microsoft\SystemCertificates\CA\Certificates";
        static void Main(string[] args)
        {
            CertificateStorePath[] paths =
            {
                new CertificateStorePath()
                {
                    Name = "Local Computer",
                    RegKey = userRegKey,
                    Hive = RegistryHive.LocalMachine
                },
                new CertificateStorePath()
                {
                    Name = "SYSTEM Account (LocalSystem)",
                    RegKey = "S-1-5-18\\" + userRegKey,
                    Hive = RegistryHive.Users
                },
                new CertificateStorePath()
                {
                     Name = $"Local User ({Environment.UserName})",
                     RegKey = userRegKey,
                     Hive = RegistryHive.CurrentUser
                }
                
            };

            foreach (CertificateStorePath path in paths)
            {
                Console.WriteLine($"Searching Intermediate Certificates for {path.Name}...");
                Console.WriteLine();
                Console.WriteLine();

                // Open the key.
                using (RegistryKey k = RegistryKey.OpenBaseKey(path.Hive, RegistryView.Registry64))
                {
                    using (var basekey = k.OpenSubKey(path.RegKey))
                    {
                        string[] subkeys = basekey.GetSubKeyNames();
                        foreach (var subkey in subkeys)
                        {
                            using (var sub = basekey.OpenSubKey(subkey))
                            {
                                var value = sub.GetValue("Blob");
                                if (value is byte[])
                                {
                                    // Load the certificate.
                                    X509Certificate2 cert = new X509Certificate2((byte[])value);
                                    Console.WriteLine($"Certificate [{subkey}]: Subject={cert.Subject}, Issuer={cert.Issuer}, Thumprint (SHA1)={cert.Thumbprint}");
                                    Console.WriteLine();
                                }
                            }
                        }
                    }
                }

                Console.WriteLine();
                Console.WriteLine();
                Console.WriteLine();
                Console.WriteLine();
            }


            Console.WriteLine("Press a key to exit.");
            Console.ReadKey();
        }


        private struct CertificateStorePath
        {
            public string Name { get;set;}
            public string RegKey { get; set; }
            public RegistryHive Hive { get; set; }
        }
    }
}

Using the registry key, I think you should be able to delete a certificate by deleting the corresponding key.
(It should be possible to rewrite this as a PowerShell script but I don’t have much knowledge of powershell.)

However, what still is strange is that

  1. why intermediate certificates are added spontaneously to the user’s intermediate certificate storage (at least this shouldn’t happen for the system account), and
  2. why SChannel is able to build a chain with the old X1 intermediate whereas other SSL tools will flag that as a chain error. I think this is a bug in the SChannel. Perhaps someone could clarify this with Microsoft?

Thanks!


#106

This guy right here… this guy… he is my HERO. I also had to remove the bindings and re-add them.


#107

Is it possible to use the “Certificates” snap-in to the MMC and just load up the “Service Account” that’s used by IIS (i.e. World Wide Web Publishing Service).

I believe you should be able to get to any cert store on the host using this method.


#108

Building on the previous info from @Knagis and @T917820981726: running the following in the command prompt as Administrator deletes the X1 intermediate cert from the SYSTEM user and Local Machine accounts, respectively. This plus manually cycling the binding away from and back to the cert in IIS Manager fixes the problem. (As @ grudnitzki also stated: this fix means replacing all older certificates depending on the X1 intermediate certificate).

reg delete HKU\S-1-5-18\Software\Microsoft\SystemCertificates\CA\Certificates\3EAE91937EC85D74483FF4B77B07B43E2AF36BF4 /f
reg delete HKLM\Software\Microsoft\SystemCertificates\CA\Certificates\3EAE91937EC85D74483FF4B77B07B43E2AF36BF4 /f

#109

2 posts were split to a new topic: IIS chain building problems returning on renewal


#110

Thanks a lot! You saved me hours of research!