SSL Certificate Chain Issues Azure Application GatewayV2

My domain is: lukeuhren.comhttps://aspen.lukeuhren.com … site is not up 247 as this is a test to try and resolve this issue right now

I ran this command:

Recently I've created a powershell script using POSH-ACME to generate, export and import my certificates into an Azure Application Gateway load balancer. The issue I have is odd with testing the SSL Cert chain on say https://www.ssllabs.com/. It said it didn't have incorrect order, then I generated a new one and then that site said it did and contains anchor. The main issue here is, that all the other sites to test SSL Certs say there is no chain issues and its 100% fine. Sites like SSL Checker , SSL Certificate Checker - Diagnostic Tool | DigiCert.com , Check SSL Certificate and https://www.thesslstore.com/ssltools/ssl-checker.php. I find it odd that all other sites show fine and ssllabs doesn't, so for some reason that doesn't sit right with me. Is ssllabs really the only one catching this?

If there is a chain issue and contains anchor, how do I avoid this? I can explain EXACTLY what I am doing as well.

I am issuing a wildcard for a specific domain that is personal for testing

$certname = ".lukeuhren.com", "lukeuhren.com"
$certsubject = "
.lukeuhren.com"
$RootCertName = "R3"
Install-Module -Name Posh-ACME -AllowClobber -Force
Set-PAServer LE_PROD
Set-PAAccount -Contact user@gmail.com -Force

#issue cert
New-PACertificate $certname -AcceptTOS -Contact user@gmail.com -DnsPlugin AcmeDns -PluginArgs @{ACMEServer='auth.acme-dns.io'} -Verbose -Install -Force

#get cert data
$certdata = Get-ChildItem -Recurse -Path Cert:\LocalMachine\My | ?{ $.Subject -Like "CN=$certsubject" -and $.Issuer -Like "CN=$RootCertName*" -and $_.NotBefore -le (get-date).AddHours(1)}

#export cert
certutil -p $certpassword -f -exportPFX My $certdata.Thumbprint c:\Scripts\wildcard.pfx

I can export other ways as well and there is no difference.. there is also a fullchain.pfx getting the location from Get-PACertificate | fl and still would say chain issues on ssllabs
#cert.cer (Base64 encoded PEM certificate)
#cert.key (Base64 encoded PEM private key)
#cert.pfx (PKCS12 container with cert+key)
#chain.cer (Base64 encoded PEM with the issuing CA chain)
#chainX.cer (Base64 encoded PEM with alternate issuing CA chains)
#fullchain.cer (Base64 encoded PEM with cert+chain)
#fullchain.pfx (PKCS12 container with cert+key+chain)

What would someone suggest command wise to export out if this is showing incorrect order and contains anchor on ssllabs? I am a bit confused why this is having an issue with how I am going about it

It produced this output:

Chain issues Incorrect order, Contains anchor

My web server is (include version):
Azure Application Gateway v2/Server 2012R2 IIS 8.5

The operating system my web server runs on is (include version):
Azure Application Gateway v2/Server 2012R2 IIS 8.5

My hosting provider, if applicable, is:
Microsoft Azure

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
Right now this is just testing the SSL cert on Azure Application Gateway Load balancer. I have full control over it

Hey @lukeuit. You seem to be doing everything right from a Posh-ACME perspective. I'm not terribly familiar with Azure App Gateway though. Is the PFX file uploaded to it directly or via Azure Key Vault or something? Do you have any control over the chain config on the Azure side?

This may end up being a quirk specifically with ssllabs and Azure App Gateway. The next time you test, I would suggesting using openssl to see exactly what chain App Gateway is sending to clients like this:

> openssl s_client -connect aspen.lukeuhren.com:443

Among the wall of text output towards the top should be a section that looks like this (assuming the default long chain):

---
Certificate chain
 0 s:CN = aspen.lukeuhren.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

This indicates that the server is sending the leaf certificate (0) issued by R3, the R3 intermediate (1) issued by ISRG root, and the ISRG root (2) issued by the DST Root.

Depending on how App Gateway does its chain building, it's possible it might leave out the ISRG Root (2). But it should at least have 0 and 1.

hey @rmbolger ... Thanks for the reply it is appreciated as I have been banging my head against the wall on this one.

With Azure Application Gateways I upload the pfx file or call the file from a key vault depending if I manually tested or deployed with what is ARM JSON template for automation deployment or powershell. I am not aware of any control over the chain config on the azure side, so I may have to ask Microsoft on this one and confirm. Supporting Azure myself, I find there is pieces they don't mention in their documentation easily or are easy to find anyway.

With that output are you saying that this is showing the correct chain to you? openssl s_client -connect aspen.lukeuhren.com:443

I even used the fullchain.pfx and uploaded it if that is recommended or not and it cleared the anchor error on sslabs but incorrect order still remains. All other sites still say this certificate is compliant on the chain.

I couldn't actually check earlier because when I tried, I think the site was down. So what was in the post was a guess. Here's what I got when I checked just now.

---
Certificate chain
 0 s:CN = *.lukeuhren.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
 2 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

So it does look like the gateway is sending the chain out of order. It's got all the right pieces, but 1 and 2 are typically served in reverse so that R3 comes before ISRG Root X1. I have no idea what that means as far as client compatibility though. I also don't really know how to approach Azure about the issue.

I'm curious what happens if you try requesting the shorter alternate cert chain. It only has one cert in addition to the leaf cert so there would be nothing to get out of order. You can check by changing your New-PACertificate call to include -PreferredChain 'ISRG Root X1'. But if you do that, I'd also recommend using the Posh-ACME created fullchain.pfx file rather than the exported version from certutil.

If it works, keep in mind that you'll lose really old Android compatibility with the shorter chain.

@rmbolger

I have followed your suggestions on the -PreferredChain 'ISRG Root X1' switch in the command ... where its .... New-PACertificate $certname -AcceptTOS -Contact test@gmail.com -DnsPlugin AcmeDns -PluginArgs @{ACMEServer='auth.acme-dns.io'} -Install -PreferredChain 'ISRG Root X1' -Force -Verbose

I then tested with the certutil export and it still said incorrect order like you would of thought. Then I used the fullchain.pfx and ssllabs now shows none to chain issues. SSL Server Test: aspen.lukeuhren.com (Powered by Qualys SSL Labs)'

So with this shorter chain, this will at least solve this for now but is it recommended to have it shorter and that only will affect old android OS?

So now it looks like...

fullchain.pfx with -PreferredChain 'ISRG Root X1'
Certificate chain
0 s:CN = *.lukeuhren.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

fullchain.pfx without -PreferredChain 'ISRG Root X1'
Certificate chain
0 s:CN = *.lukeuhren.com
i:C = US, O = Let's Encrypt, CN = R3
1 s:O = Digital Signature Trust Co., CN = DST Root CA X3
i:O = Digital Signature Trust Co., CN = DST Root CA X3
2 s:C = US, O = Let's Encrypt, CN = R3
i:O = Digital Signature Trust Co., CN = DST Root CA X3

As for application gateway chain order I will have to look into it and ask Microsoft as I have premier support if they know. It could go nowhere and take a while as many of their support people may not even know

Thanks again for this and excellent community help here!

Shorter is generally better because there's less data to transfer over the wire. This is also a reason why some folks prefer ECC over RSA certs (they're smaller). However, your target audience and their client compatibility should likely take priority over the bandwidth savings from a shorter chain.

It will definitely affect old Android (pre 7.1.1) clients. It would also affect any other client that doesn't trust the ISRG Root X1 certificate which became widely trusted around 2016. The most likely affected client outside of old Android stuff would likely be old IoT devices that don't or can't get updates. But if those types of devices won't be connecting to your App Gateway, it shouldn't matter.

Perfect, thanks for the explanation and help. I believe that's all I have then for questions and seems to have solved my issue then. I will update if I find anything regarding application gateway chain ordering in Azure from Microsoft in case anyone else finds this post for help

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.