Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is: fmp.com
My web server is (include version): Apache/2.4.18 (Ubuntu)
The operating system my web server runs on is (include version): "Ubuntu 16.04.7 LTS"
I can login to a root shell on my machine (yes or no, or I don't know): yes
The version of my client is (e.g. output of
certbot --version or
certbot-auto --version if you're using Certbot): certbot 1.32.1
First, I know I'm running an outdated version of Ubuntu server, however security fixes are supplied by Ubuntu ESM so I should be good for a few years now (a version upgrade will require SERIOUS reconfig of a number of customer service subsystems).
I run the Ubuntu package update system (aptitude) on my server several times a week to keep things up to date. Today, the ca-certificates package got updated from 20211016~16.04.1~esm1 to 20211016~16.04.1~esm2 and customer mail systems got hit with TLS errors when trying to both retrieve and send email through the server and all hell broke loose!
I have good backups, and replaced the /etc/ssl/certs folder with one from last week, which solved the immediate problem, but this is just a band-aid.
A diff -u on the old certs directory shows that several certs were removed from the ca-certificates.crt file in the upgrade, all of them issued by TrustCor Systems. The OU and CN on these are:
OU=TrustCor Certificate Authority, CN=TrustCor ECA-1
OU=TrustCor Certificate Authority, CN=TrustCor RootCert CA-1
OU=TrustCor Certificate Authority, CN=TrustCor RootCert CA-2
diff also shows the following files missing from the updated /etc/ssl/certs folder:
Only in certs: d0cddf45.0
Only in certs: TrustCor_ECA-1.pem
Only in certs: TrustCor_RootCert_CA-1.pem
Only in certs: TrustCor_RootCert_CA-2.pem
So it seems that TrustCor certs were eliminated altogether in this upgrade. I seem to recall that there was an issue with Lets Encrypt's certs trust chain some time ago and that TrustCor was picking up the slack, but I've forgotten how I addressed this previously.
What do I need to do to bring trust chain lookups back into compiance going forward so that my server's mail won't break going forward, and I won't have customers screaming at me?