New Slack certs rejected by PaloAlto PanOS firewalls

Slack.com updated their certs yesterday (3/14/2023) and our PaloAlto firewalls immediately began rejecting them as invalid.

From what I can see in openssl s_client (see below), it looks like their X1 root is signed/issued by X3 (expired) instead of being self-signed. Is this cert chain invalid or am I misunderstanding?

** **% openssl s_client -connect hooks.slack.com:443** **CONNECTED(00000005)** **depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1** **verify return:1** **depth=1 C = US, O = Let's Encrypt, CN = R3** **verify return:1** **depth=0 CN = slack.com** **verify return:1** **---** **Certificate chain** ** 0 s:/CN=slack.com** ** i:/C=US/O=Let's Encrypt/CN=R3** ** 1 s:/C=US/O=Let's Encrypt/CN=R3** ** i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1** ** 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1** ** i:/O=Digital Signature Trust Co./CN=DST Root CA X3** **---** **

surprised they didn't fixed for long chain yet?

3 Likes

Since our problem is not in a decryption profile on the palo but rather in a Log Forwarding profile, I don't know that this change would apply to us, but I will look.

But is the fact that slack's chain seems to show X1 issued by X3 wrong? On other letsencypt cert servers, I see X1 signed by X1 (self-signed) which is what I expected would be the right chain.

No, that's the so called "long chain" which is the default chain currently. Please search this Community for more info.

3 Likes

Ok then on my firewalls, I have R3 and X1 (self-signed) loaded and trusted, but it does not validate. the Palo will not let me load the X1 (cross-signed X3) since it is expired.

So I'm up a tree.

Half the sites on the Internet probably use this "long chain" (including this community forum you're using now). Whatever issue you're having, it's either not related or you'd be having problems with a lot more than one site.

6 Likes

Or even better: change firewall manufacturer.

2 Likes

Here are some forums that maybe a better place to search for solutions

And of course
https://support.paloaltonetworks.com/Support/Index

2 Likes

Also @griffin posted Long (default) and Short (alternate) Certificate Chains Explained

3 Likes

https://www.ssllabs.com/ssltest/analyze.html?d=hooks.slack.com

If you expand the "Certification Paths" section, you will see it shows two trusted paths.
One of the two [if not both] should work with Palo.
If neither work, then open a ticket with Palo about that.

3 Likes

Aye, it looks like long-chain creates validation error for the Palo. My simple test:

if I select la-sso.bounce51.com and run the Palo's connection test it returns OK -- this is a server with the short chain letsEncrypt.

when i use hooks.slack.com and run the Palo's connection test it returns cert error.

when i use letsencrypt.org and run the Palo's connection test it returns cert error.

I am running it down with Palo. Hopefully they have a solution.

2 Likes

Just for fun, here's some other long-chain Let's Encrypt users (I didn't check they all use long-chain but all the biggest I checked do)

3 Likes

Yep, when I point the palo at any of those LE long-chain servers, it says no-way-Jose.

1 Like

In light of its being a Palo Alto firewall, maybe it should be "no-way-San-Jose"? :slight_smile:

We knew in the past that some devices would have a problem with the long chain, but I think this is the first time someone has reported here having a Palo Alto firewall reject all of them. Could you try asking about it on the Palo Alto forum and see what that community has to say about it?

4 Likes

Yessir, I've posted the question in the PAN forum and have a support case open with them.

Their forum appears to be... less active than this one, to say the least. I appreciate the quick & accurate responses from everyone here. Really helping me to understand more about the situation, and hopefully will help me to teach Palo what they're doing wrong.

5 Likes
2 Likes

What's wrong with this solution posted earlier?

3 Likes

That solution is for the SSL decryption system (where you MITM attack your users so you can inspect their encrypted traffic). It doesn't apply to the Log Forwarding service which is the firewall itself just sending traffic/threat/system logs to a webhook.

4 Likes

Why is your log forwarding service sending traffic/thread/system logs to slack.com?

2 Likes

That's pretty common nowadays (at least for events that may require someone to act on them). Sort of a substitute for a pager service.

4 Likes