Struggling to get new ISRG root certificate to be recognized in Ubuntu 16

Help
1

I have two separate servers, one for self-hosted mail (with Mail-in-a-box) and one for my web server (with an application that uses PHPMailer), both running Ubuntu 16.04 and using LE. Like many others, I got bitten when the old DST Root CA X3 certificate expired last month. Specifically, my mail server began rejecting outgoing mail from the client - in this case my web server. I found the postfix error in my mail logs:

warning: TLS library problem: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired

and after some googling eventually traced it back to the root cert expiring (see Let's Encrypt's Root Certificate is expiring!).

I tried a few things, including:

  • Updating the version of openssl on my web server (from 1.0.2g to 1.1.1k)
  • Checking the version of openssl on my mail server (it's also running 1.1.1)
  • Removing the old DST certificate from my web server's /etc/ca-certificates.conf list and rebuilding with dpkg-reconfigure -fnoninteractive ca-certificates and update-ca-certificates

Now I get a slightly different error from postfix on my mail server:

TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca.

I tried running the following command on my web server:

openssl s_client -connect mymailserver.com:587 -servername mymailserver.com -starttls smtp

And I get the following:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = mymailserver.com
verify return:1
---
Certificate chain
 0 s:CN = mymailserver.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Is this a problem on my web server, mail server, or both? What else do I need to do to fix it?

1 Like
2

Hi @alexweissman and welcome to the LE community forum :slight_smile:

That is actually as good as it can get (using the long/default chain).
OpenSSL is showing "`verify return:1" on all.

If you don't service any older Android systems, you could try switching to the shorter/alternate chain.

3 Likes
3

Why not run that interactively?
dpkg-reconfigure ca-certificates
You would get to see which certs are there and choose which to include/exclude.

3 Likes
4

Thanks for your help. Which certs should I exclude to force openssl to use the shorter chain? It looks like DST_Root_CA_X3.crt is still excluded according to the ca-certificates tool, but I need to exclude certain intermediate certs too right?

1 Like
5

No, only exclude "DST Root CA X3".
And ensure "ISRG Root X1" is included.

4 Likes
6

Hmm, according to the tool I've correctly excluded the old cert and included the new one:

Processing triggers for ca-certificates (20210119~16.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

I saw a few recommendations to run c_rehash in my certs directory, so I tried that:

Doing /usr/lib/ssl/certs
WARNING: dhparam.pem does not contain a certificate or CRL: skipping
WARNING: Skipping duplicate certificate ca-certificates.crt
WARNING: Skipping duplicate certificate ca-certificates.crt

I feel like there must be something wonky or unusual about my configuration, especially since I've been using LE on this server almost from the beginning of LE. Or, there is something super obvious that I need to be doing that I haven't.

I also read something about cross-signed vs non-cross-signed versions of the ISRG cert. Is it possible that my web server and mail server have different versions of this cert, and that's why it's being rejected on the mail server? I tried running:

openssl x509 -text -in /etc/ssl/certs/ISRG_Root_X1.pem

on both machines, and they appear to be identical.

1 Like
7

Wait! It seems to be working now (i.e. my web server can now negotiate the TLS and authenticate with my mail server). It seems that just letting it sit overnight allowed it to find the alternate chain? Is there some kind of cert caching going on with openssl and Ubuntu that just needed to refresh?

At any rate, I appreciate your help!

2 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.