Upgrade of Ubuntu ca-certificates package breaks cert trust chain [SOLVED]

Help
#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: fmp.com

My web server is (include version): Apache/2.4.18 (Ubuntu)

The operating system my web server runs on is (include version): "Ubuntu 16.04.7 LTS"

I can login to a root shell on my machine (yes or no, or I don't know): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.32.1

First, I know I'm running an outdated version of Ubuntu server, however security fixes are supplied by Ubuntu ESM so I should be good for a few years now (a version upgrade will require SERIOUS reconfig of a number of customer service subsystems).

I run the Ubuntu package update system (aptitude) on my server several times a week to keep things up to date. Today, the ca-certificates package got updated from 20211016~16.04.1~esm1 to 20211016~16.04.1~esm2 and customer mail systems got hit with TLS errors when trying to both retrieve and send email through the server and all hell broke loose!

I have good backups, and replaced the /etc/ssl/certs folder with one from last week, which solved the immediate problem, but this is just a band-aid.

A diff -u on the old certs directory shows that several certs were removed from the ca-certificates.crt file in the upgrade, all of them issued by TrustCor Systems. The OU and CN on these are:

OU=TrustCor Certificate Authority, CN=TrustCor ECA-1
OU=TrustCor Certificate Authority, CN=TrustCor RootCert CA-1
OU=TrustCor Certificate Authority, CN=TrustCor RootCert CA-2

diff also shows the following files missing from the updated /etc/ssl/certs folder:

Only in certs: d0cddf45.0
Only in certs: TrustCor_ECA-1.pem
Only in certs: TrustCor_RootCert_CA-1.pem
Only in certs: TrustCor_RootCert_CA-2.pem

So it seems that TrustCor certs were eliminated altogether in this upgrade. I seem to recall that there was an issue with Lets Encrypt's certs trust chain some time ago and that TrustCor was picking up the slack, but I've forgotten how I addressed this previously.

What do I need to do to bring trust chain lookups back into compiance going forward so that my server's mail won't break going forward, and I won't have customers screaming at me?

#2

Can you provide the chain(s) in use?

2 Likes
#3

How can I determine the chain(s) in use? I can easily read the crt and pem files, but how does this fit together to answer your question?

#4

Identrust, not TrustCor. Let's Encrypt has never relied on the latter.

5 Likes
#5

To my best knowledge, Let's Encrypt has never had any type of relationship with TrustCor. Anything related to TrustCor is not related to Let's Encrypt.

While this has nothing to do with Let's Encrypt, I am actually aware of the TrustCor removal. This is a distrust event from Mozilla, which was decided recently after concerns were raised about TrustCor. You can find more information in this MDSP thread.

We don't know, as we have no idea what's causing your issues.

(I do see the possibility that if you have manually removed DST Root CA X3 from your trust store it might have come back with the update - but you haven't mentioned anything in this regard so far, so :man_shrugging:)

4 Likes
#6

Well I posted the diffs in the /etc/ssl/certs file. Normally, they are symlinked to files in /usr/share/ca-certificates/mozilla/ but my band-aid fix pulled the entire certs folder w. symlinks dereferenced, replaced by the real files.

The diff readout shows TrustCor certs missing, as copied from openssl displays of the diff output, and "missing files" notices. I'll accept your word on this. I just need to know how to fix the problem.

#7

How do I determine the trust chain in use?

If I re-enable the updated cert folder, how can I determine where the trust chain breaks?

I HAVE to fix this problem, even if we have to hire a consultant to do it.

#8

The most useful information is going to be information about the errors:

  • What the environment of the TLS/email client that produces the error is, like operating system version and OpenSSL version.
  • What the contents of the error message is
  • What the TLS server hostname is (is it linode.fmp.com?).

The SMTPS and STARTTLS SMTP certificates on that server seem fine and have a correct and widely compatible chain. We don't really have the right information to troubleshoot this.

2 Likes
#9

I've removed nothing. I just ran Ubuntu CLI aptitude to upgrade the ca-certificates package and the system broke.

#10

This one went unexplained:

3 Likes
#11

I checked the hash and that's just one of the TrustCor certs (TrustCor Certificate Authority, CN=TrustCor RootCert CA-2) again, so not related.

4 Likes
#12

Yes, I restored the /etc/ssl/certs folder from a backup on an in-house system running a more recent version of Ubuntu. The symlinks in the cert directory were de-referenced and replaced with pem files from the backup server.

So if you've looked at my system, yes, it's working now, and probably looks normal, but that's because I did what I had to to get it up and running for customers.

It's after business hours now. I can replace the ssl/certs folder in use with the "updated" one which broke mail, and if you can poke at it and see what the issue may be, it may help. Let me know.

#13

Then there is more to this than meets the eye.

What else may have been altered [and restored]?

2 Likes
#14

Correct. It's a symlink chain of reference to a file in /usr/share/ca-certificates/mozilla which is no longer there.

#15

Nothing else was explicitly altered. I did renew the Courier (mail server) .pem files using certbot, but there was nothing else changed. I only did this AFTER I started getting customer complaints that mail wasn't working.

#16

Basically, everything in ssl/certs got replaced in the restore. The borked ssl/certs was a collection of symlinks to files in /usr/share/ca-certificates/mozilla on FMP's publlic server. When I pulled the backup off of the backup server on our LAN, the symlinks in the certs file were de-referenced and replaced with LOCAL .pem files from the backup server. The misbehaving certs folder was manually moved to /ssl/certs.o so it's still there. A cursory look at it indicates that it's a collection of symlinks as I noted, and for the purposes of analysis, it could easily be moved back into place.

#17

Take a moment. The best solution here is to go at it from the client point of view, what is connecting to where and what problem does it have with the certs. This is better than trying to restore a broken and outdated CA store.

Typically you can use openssl s_client -connect <hostname>:<port> to review the TLS response. Ping Identity Documentation Portal

It also matters what the client is and how updated they are. If they don't know ISRG Root X1 (self signed) for example. On some operating systems/clients the chain you present from the server matters, on others (typically windows) it builds it's own from the end-entity cert and it ignores the rest of the chain you serve.

4 Likes
#18

Clients are varied. I'm using evolution (Linux). The customers who called about the problem are using MS Outlook. My wife, who saw the problem on her business computer, is running Thunderbird for Windows 10.

The error message in my mail client basically reported a failure of TLS in the mail capabilities IMAP exchange. I don't have it exactly.

Everyone uses mail.fmp.com for mail transactions.

Yes, at this point it would. If you are of a mind to look at it, I'll restore the misbehaving certs folder for a while. Let me know. It broke both SMTP and IMAP, both of which use Let's Encrypt certs.

1 Like
#19

I moved the problem certs folder back into place. The error I'm getting from evolution is "The reported error was “Failed to get capabilities: Error performing TLS handshake: An unexpected TLS packet was received.”".

openssl s_client -connect mail.fmp.com:143 yields ...

CONNECTED(00000003)
140157591254336:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 304 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
-

The error appeared quite suddenly across numerous mail clients both for IMAP and SMTP. Error reports are similar. I currently have the misbehaving ssl/certs folder in place but can't leave it this way.

#20

For what it's worth, with openssl v3.0.2 I got this

openssl s_client -connect mail.fmp.com:25 -starttls smtp

CONNECTED(00000003)
40B77F2D0F7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:308:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 229 bytes and written 354 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
3 Likes