New Slack certs rejected by PaloAlto PanOS firewalls

Curious if anyone can advise why openssl s_client -connect letsencrypt.org:443 is showing non-LetsEncrypt certificate? But in my browser I show LE R3 -> X1

For me, it doesn't?

2 Likes

About Let's Encrypt - Let's Encrypt read about Internet Security Research Group (ISRG)
And here lencr.org - Let's Encrypt

I see this

$ openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
$ openssl s_client -connect letsencrypt.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = lencr.org
verify return:1
---
Certificate chain
 0 s:CN = lencr.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Feb  2 00:00:24 2023 GMT; NotAfter: May  3 00:00:23 2023 GMT
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
---
Server certificate
1 Like

Got any "security" devices/software intercepting traffic between you and the Internet? What do you actually see?

Also, it doesn't matter in this instance, but generally you should also include -servername <blah> in the openssl command to send a host header with the request. Some web servers will serve different certs based on that.

5 Likes

On up to date OpenSSL versions that isn't required: s_client will always send the server name from the hostname, unless overwritten by the -servername option.

6 Likes

Also related to this thread's Topic is Verifying a certificate - #5 by jsha

1 Like

TIL

Is there a way to explicitly not send a host header for the rare cases you actually want to test that?

3 Likes

Well, Host: is different from SNI (HTTP layer rather than TLS layer). In fact, sending different values in the two protocol layers has been used as part of an anti-censorship technique called domain fronting.

In newer OpenSSL, you can avoid sending SNI with the -noservername option:

       -noservername
           Suppresses sending of the SNI (Server Name Indication) extension in
           the ClientHello message. Cannot be used in conjunction with the
           -servername or <-dane_tlsa_domain> options.
8 Likes

As noted, your openssl version is probably old and not sending the server name (1.0.1k for example does not by default).

letsencrypt.org is served by a CDN. A CDN needs to know the server name (domain) to direct to the proper origin server. In this case, the CDN uses a default DigiCert when none is supplied.

5 Likes

What cert do you get?

3 Likes

If I use -noservername (like older versions of s_client do by default), I get one with

subject=C = US, ST = California, L = San Francisco, O = "Netlify, Inc", CN = *.netlify.app

(which is not too surprising to me for the CDN reason that @MikeMcQ alludes to above)

5 Likes

Thanks guys. Using -servername does return correct server. Apparently macOS monterey's openssl is old. I could have sworn that I recently got LE details without -servername but must have been from a different OS.

I'm still fighting the fight with Palo and Slack. Someone will give.

2 Likes

Maybe you could trick it... into submission!
Setup a inverse/reverse proxy to hide the slack site from Palo's direct sight.
[this may require using DNS/proxies/NAT/etc. in ways that may reshape the time continuum - LOL]

2 Likes

You can check with openssl version.

1 Like

Could be connecting to a server (not a CDN) that only has a Let's Encrypt certificates and no others, in which case the default no-SNI certificate would be the one issued by Let's Encrypt.

4 Likes

Could also be remembering things, not as they were, but as they should have been - LOL

3 Likes

That would cut off a giant chunk of the internet's users, with disproportionate effects based on class and geopolitical regions.

The history of the "Long Chain Solution" includes most of the browser, os, and SSL projects coming together to loosen restrictions and coalesce around making the "short circuit" chain validation logic the standard default. A lot of key players in the greater internet industry came together for this solution. It is unfortunate that Palo Alto decided not to join everyone else.

IMHO, Palo Alto makes subpar products without much thought or foresight. They are notorious in this forum (as some comments above indicate) for randomly implementing rules that block authentication without adequately notifying their customers, if at all.

7 Likes

Will those users be perpetually permitted to use the expired / unsafe cert? Or were they just given extra time before they will actually be cut off in 2024?

It's not unsafe.

3 Likes

Apologies if I mis-spoke. Why do root certs expire?

1 Like