Yeah I guess they maybe rebuilding the Cpanel openssl 1.0.2 package with X509_V_FLAG_TRUSTED_FIRST flag enabled by default
4 Likes
In the end the only thing that matters is the question: @meisner Do you require Android support for versions older than 7.1.1? If so, you'd need to wait until cPanel has fixed their panel so you can upload the long chain including the ISRG Root X1-signed-by-DST Root CA X3. If not, just use the shorter chain with just the R3-signed-by-ISRG Root X1 intermediate.
4 Likes
Not sure if any of my clients are using older Android. I've been using the short chain and nobody has complained, so I'm guessing none of my clients are on older Android - or at least it's not very prevalent.
I think my confusion just centered around always using the ca.cer that was provided and assuming everything would be fine using that. I just assumed the chain was being sent by Let's Encrypt and it was the proper chain to use. Apparently that's not necessarily the case.
One interesting note though. I've also got a DirectAdmin server. It uses the same acme.sh client to issue certificates and uses the DirectAdmin API to install those certificates. With DirectAdmin, the long chain works just fine. IMAP, SMTP, Web all work fine with this long chain on DirectAdmin.
So in regards to the long chain failing OpenSSL verification - why is that not also failing with DirectAdmin? Or is this a case of cPanel doing too much?
When it comes to managing a server, DirectAdmin has always seemed more of a: "OK, do what you want, we assume you know what you're doing."
Where as cPanel has been more of a: "You're probably stupid, so let us handle this for you."
3 Likes
As an ACME client developer and longtime cPanel user myself, I can say "yes it is."
4 Likes
That's kind of what I thought too. And that's kind of what pisses me off.
I think my system of using acme.sh to issue certificates and then install the information that acme.sh provides should have resulted in installation of the certificate. Regardless of whether the short chain or long chain was being used.
But instead something was done in cPanel that prevented the long chain from being installed and that broke IMAP/POP3/SMTP secure services for my clients.
Maybe the long chain technically isn't advised. But as long as the IMAP/POP3/SMTP/Web client is up-to-date, it shouldn't really matter if the long chain is used. If the IMAP/POP3/SMTP/Web client isn't up-to-date then maybe the long chain helps, maybe it doesn't.
I don't know if you necessarily lose anything by using the long chain - but you do potentially gain something.
3 Likes
As an aside, I bought a Samsung Galaxy Tablet S2 (9.7) less than 4 years ago. It is stuck at Android 7.0 so needs the long chain to avoid the 'site is not secure' message. I tried it yesterday given these sorts of discussions - I don't often use this tablet. I was surprised a recently purchased device was 'old'.
7 Likes
4 years, what do you expect? In today's Android phone manufacturer world, you're not allowed to advertise a device older than 6 months. After 2 years it's dead, only some "premium" manufacturers give you 3 years of updates. Then you can throw away the whole thing and buy a new one, even though the microchips are perfectly fine. This is such a waste...
Also, the Samsung Galaxy Tab S2 went out of software support in 2017, so it was likely already long dead in the water when you bought it. Even more absurd is that Android 8 was released less than two years after Samsung releases the Tab S2, but Samsung still did not bother to update the thing - they did not even patch the phone to 7.1 (which came out in 2016!). That means you're stuck with a lot of 7.0 bugs, that were fixed in 7.1. It's just absurd. (It's probably related to the internal two-major-updates-only policy Samsung followed back then, and since it already had updates to the 6 and 7 series, it was dropped)
6 Likes
Agreed.
I bought it new. I have a variety of tablets so am not bothered. My point is more that it was not that long ago. Plenty of people still use such devices not wanting to incur the cost and/or aggravation of replacing them.
When choosing which chain to serve, the short chain leaving off 'old' Android isn't that old of a device. Regardless of what manufacturers want. The number of such devices still in operation is large. Probably why Lets Encrypt suggests serving the long chain. And, why IdenTrust recently made similar move.
Is this a case of us being in 'violent agreement'? 
4 Likes
Actually IdenTrust made their short chain the default and their long chain the alternate.
6 Likes
I guess I'm further confused by cPanel's claim that the long chain doesn't verify with CentOS 7 openssl.
Where is this verify taking place?
The DirectAdmin server uses Dovecot and Exim just like the cPanel server. And the DirectAdmin server's Dovecot and Exim are handling the long chain just fine. Perhaps there are differences in how each of these are built between DirectAdmin and cPanel, that could be true (but I kind of doubt it, at least in regards to this). Both servers are CentOS 7 using OpenSSL 1.0.2k-fips.
So, when cPanel says it's not verifying I have to assume it's during the install process. Where you paste in the certificate, paste in the key, and paste in the long chain. When you click Install, cPanel says "You're probably stupid and don't really know what to put in the ca bundle field, so we're going to check... NOPE! that one's bad, I ain't installing anything!"
And the DirectAdmin when you click Install "Verify.. Shermify.. you put it in there, we assume you know what you're doing, if it happens to be wrong the certificate won't work and that'll be your cue that something isn't right. Not our place to be your babysitter."
2 Likes
That version of openssl needs the -trusted_first flag to correctly verify the long chain. That flag is essentially the default in openssl 1.1 so this makes 1.0.2k work similarly to that.
Update: nevermind. I see this was brought up earlier in the thread
4 Likes
They're probably calling openssl verify (or similar) internally before installing the certificate, and that indeed doesn't handle the long chain well. This is true especially for older openssl versions. When serving chains (as a server), openssl's SSL/TLS stack doesn't check the validity of the chain, so any openssl version will handle this just fine. Problems only occur when verifying certificates (e.g as a TLS client).
5 Likes
That's right - I forgot - my bad. Still, they are taking action to accommodate 'old' android.
3 Likes
You have the inside track on that discussion.
3 Likes
Actually port 587 with STARTTLS is currently not recommended anymore, and port 465 was "un-obsoleted" and now recommended by RFC 8314 - Cleartext Considered Obsolete: Use of Transport Layer Security (TLS) for Email Submission and Access (section 3.3).
Same for POP3 and IMAP, currently recommended (again) on their "implicit TLS" ports 995 and 993 instead of discouraged STARTTLS on ports 110 and 143.
STARTTLS has had several implementation vulnerabilities, both client-side and server-side, and is generally to be avoided these days: https://nostarttls.secvuln.info/
8 Likes
It's been a while since I've played with cPanel. But just tried AutoSSL with Letsencrypt and cPanel SSL certs. Seems cPanel SSL certs are Comodo/Sectigo based like ZeroSSL and both have alternate chain path to AAA Certificate Services crossed signed CA root for older device compatibility. So not sure what the issue is. If you want older device compatibility, switch to cPanel Comodo/Sectigo based SSL certs. Otherwise, for others either choose Letsencrypt or cPanel Comodo/Sectigo based SSL certs!
2 Likes
cPanel does indeed do an OpenSSL verify prior to certificate installations. This has been so for a long time as a means to prevent âfoot-gunningâ. cPanel also, in the same vein, ensures that the cert matches at least one domain on the associated web vhost.
This DST root expiry exposed a need to reconfigure OpenSSL to use trusted-first verification; an update to that end is forthcoming. As a temporary fix, all AutoSSL Letâs Encrypt certificate installations on cP are using LEâs short chain for now.
3 Likes
Of note: Letâs Encryptâs CA Issuers URLs yield the short chain, so if you install just the certificate (with no CA bundle) to cPanel youâll get the short chain.
3 Likes
One anomaly in all of thisâwhich I havenât seen discussed muchâis that OpenSSL 1.0.2 willâby default!âverify LEâs longer chain in connection mode, e.g., with openssl s_client.
So apparently itâs really just the standalone verifications that need to be updated to use trusted-first verification. Thus, even OpenSSLâs own âworkaround #2â is inaccurate, as it suggests that s_client needs the trusted-first option to be configured manually.
Example:
> openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
> echo | openssl s_client -connect helloworld.letsencrypt.org:443 -servername helloworld.letsencrypt.org
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = helloworld.letsencrypt.org
verify return:1
---
Certificate chain
0 s:/CN=helloworld.letsencrypt.org
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFOTCCBCGgAwIBAgISBMKdCVsRIHqesWZmqnyMJ5EUMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA4MjUxNTAwMjBaFw0yMTExMjMxNTAwMTlaMCUxIzAhBgNVBAMT
GmhlbGxvd29ybGQubGV0c2VuY3J5cHQub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA0Zrh+0eVUDbsViovSUj1ZmvT9bhGIWe4yiEtZKaYjchyqIYi
+iuil+X7qmTDpkFE2B1UQuoqJl7XQ5J6oJ+9WsvsXU6ZcinoF+mGOLshuEaemAOF
lez2HAkEYI1YoYyiFCUGgyuSspHsHaFbuckuOn6ycv8CvPlp4E2fIjsAB/pfA9wm
fF6i4QVSBq9oEaeN5f/KcUY2jfVCTnrXfLIdWYXPJzLWYm7uiM+rwqjiUD3Hit5X
YHicGQDQC/yXoEUhNX8PIwofnag2sqZpCUuWroJDlIBNXpF/jZEO5i1RBTiUsBUt
ZqhJBXZw+LHMmfNGBdl5SUBI/QpZP1xWh70g7QIDAQABo4ICVDCCAlAwDgYDVR0P
AQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMB
Af8EAjAAMB0GA1UdDgQWBBSr158SiEGpyyqRdNU+W9eyz5dpSTAfBgNVHSMEGDAW
gBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUH
MAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3Iz
LmkubGVuY3Iub3JnLzAlBgNVHREEHjAcghpoZWxsb3dvcmxkLmxldHNlbmNyeXB0
Lm9yZzBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsG
AQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMGCisGAQQB1nkC
BAIEgfQEgfEA7wB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAAB
e34KeDgAAAQDAEcwRQIhAIQbwvcgb0Kce0VS2MqIpCAlHzCa4eo3HjI4LF8gDa92
AiBDBNsS3GQfzYXqv/8lf8KIKiPevO+/Z+U8XMvaIODxTAB1AJQgvB6O1Y1siHMf
gosiLA3R2k1ebE+UPWHbTi9YTaLCAAABe34KejMAAAQDAEYwRAIgE8AW3FbHMVIZ
mDhxx5/Q/BqMjrPJok7v/pp741aBNR0CIEH2rYN8JQcannSQrCPi9MluzYkhI4Gp
yn+u0Exmhaa1MA0GCSqGSIb3DQEBCwUAA4IBAQAJ5tuXxcIuKSZQq9Vvz8hnuo/G
Umvt90wchayidUtCRRyKAs++fT/b92a4AHSIVYWTSWEp2FzUuzGHkH4sHirKGsfN
TWAYAr9JD0bLze7/abLzhkIOcLfTv6YUAchMhNzriBCdrVztI9MGbFVHL39ldg1t
9OlkqLYNtHSaJttKktZy1R1Abnrb3AuVOuI3UKF5jctg7OOLdAcExN3LlUmD2GAY
iEfCwq+6rKljOpc1bV3knJq+C8VgMk9Ws8yofSFEzD7vvMZgT2U/Rfn45u/f3MRL
xRVMXZWX61Lz0yPlezj5Ef69FwP66ivDcLZBZIGJSmi52pNuSYwwu8rB/fXR
-----END CERTIFICATE-----
subject=/CN=helloworld.letsencrypt.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4744 bytes and written 450 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 10B1CEDEE09D4FAD7AAF689AD48A3FD3A01288661E8985ECD7EF230509710DCD
Session-ID-ctx:
Master-Key: 1131C18948B58C04EA13F4313453880CF237935FCE656718313BD2242C00EB92F5F0BC060BE2535BDC4530ED73F5DD39
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 2f e2 41 e8 bd 32 36 4d-95 bf 83 fe 38 5c 10 7a /.A..26M....8\.z
0010 - fe 44 f1 ba 37 66 9d cb-86 84 b5 22 08 c6 be 7d .D..7f....."...}
0020 - fb 67 fa ff 61 e5 3c 89-17 87 40 4b 79 00 31 67 .g..a.<...@Ky.1g
0030 - 27 1f 32 49 fb 2b b9 67-3e ef 3c 51 81 31 4d 9f '.2I.+.g>.<Q.1M.
0040 - 1c 8d c7 f1 99 ff 6f e6-26 63 11 1b b9 36 a6 b2 ......o.&c...6..
0050 - 28 df f4 b3 a4 b5 a3 dc-50 b6 8c d4 16 51 b2 7f (.......P....Q..
0060 - a6 31 fc 73 e9 90 c9 a8-5f e2 ac b2 35 8f b7 d8 .1.s...._...5...
0070 - 97 72 f3 a4 e0 b0 ee 17-89 9a 55 05 9a 67 39 8d .r........U..g9.
0080 - f8 23 e1 a8 20 94 fc 75-d9 74 d7 d4 12 53 71 a8 .#.. ..u.t...Sq.
0090 - 12 db dc 9d 3e 49 c9 6f-78 a8 70 cb 1a e8 ee fc ....>I.ox.p.....
00a0 - 4d 9e 46 5d 33 16 c6 07-10 30 3b e2 db de 24 d7 M.F]3....0;...$.
00b0 - 71 a4 ce cf b0 6d 54 f0-a6 03 4c af 48 2f b2 a0 q....mT...L.H/..
00c0 - 6c d8 23 ab 3b 96 75 84-b2 11 59 0e de ce de dc l.#.;.u...Y.....
Start Time: 1633825571
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
DONE
2 Likes
@FGasper Hmmm. I do not get the same s_client result:
openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
echo | openssl s_client -connect helloworld.letsencrypt.org:443 -servername helloworld.letsencrypt.org
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
0 s:/CN=helloworld.letsencrypt.org
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
I routinely use the -trusted_first to verify successfully.
I am running a freshly spun up Amazon Linux 2. I do not see anything to explain that in my default openssl.cnf. My openssl behavior matches the openssl blog.
3 Likes