Can't verify certificates issued by letsencrypt

I know we do have our own local CA that we had setup for self-signing certs. But I'm not sure how to verify if it got installed into ca-certificates or not. I thought we were just referencing it manually when needed.

There are still 969 bytes unaccounted for:

 200082
-199113
=======
    969

It's empty now, but I did have two in there before when the file size was 202781.

apt update
Hit:1 https://deb.nodesource.com/node_10.x bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 163 kB in 1s (301 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
4 packages can be upgraded. Run 'apt list --upgradable' to see them.
apt-get update
Hit:1 https://deb.nodesource.com/node_10.x bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 163 kB in 1s (300 kB/s)
Reading package lists... Done

A post was split to a new topic: Unable to verify certificate

Just for clarification (for both @andyrue and @rg305):

While the ca-certificates package (on Debian/Ubuntu) ships a bunch of inbuild root certificates (from Mozillas trust store), it is possible to configure all trust with this package, e.g you can add/remove (custom) certificates. It also supports running hooks whenever the trust store is updated, which may also modify the trust store further (due to applications inserting themselves there).

Configuration (auto-update, adding/removing certificates) can be done via

sudo dpkg-reconfigure ca-certificates

Manual configurations may outlive standard package reinstalls, as configuration is usually preserved by default (purge may behave differently).

A size difference of the bundle file /etc/ssl/certs/ca-certificates.crt is nothing to worry about, if all wanted root certificates are present. The file is automatically updated by the above dpkg script. New root certificates may or may not be added automatically, depending on configuration.

3 Likes

If anyone got here regarding the Letsencrypt DST Root CA X3 expired (september 30):

DO NOT RUN apt purge ca-certificates
This will literally remove certbot from your system and all ssl certificates generated by certbot.
If you have multiple vhosts on yout machine using letsencrypt as certificate, running this command is going to break all of them, making it unable to even restart your apache, which was my case.

I unfortunately runned it, but thanks to my coleague, we've been able to revert the situation by firstly disabling all vhosts using a2dissite, then reinstalling certbot, update ca-certificates, and then regenerate all ssl certificates using certbot. Quite a headache.

To fix the original problem on my ubuntu 16.04 machine, i just followed the steps mentioned on this thread:

To do that, first check if your certbot version is < 1:

sudo certbot --version

if so you have to remove it and reinstall using snap:

sudo apt-get remove -y certbot python3-certbot-apache
sudo snap install certbot --classic
sudo ln -s /snap/bin/certbot /usr/bin/certbot

After reinstalling, or If your certbot version is > 1, force the renewal:

sudo certbot renew --force-renewal --preferred-chain "ISRG Root X1"

If you're using mysql and you are having memory problems trying to install snap, try to disable mysql, install it, and then enable it again.

2 Likes

Please note that forcing renewal IS NOT NECESSARY but only adds to the already increased load on the Let's Encrypt servers. While I agree that this is the most easy way to change the certificate chain and certbot lacks proper handling of alternative chains, it's probably better to just remove the bottom certificate from fullchain.pem and chain.pem (edit both just to be sure) and reload your webserver. Does exactly the same thing without wasting precious resources on the Let's Encrypt API.

1 Like

What did you purge?

1 Like

Do you know how to reset all that and get the latest ones properly? My server is unable to connect to Letsencrypt sites.

wget https://letsencrypt.org/certs/isrgrootx1.der
--2021-10-02 00:10:13--  https://letsencrypt.org/certs/isrgrootx1.der
Resolving letsencrypt.org (letsencrypt.org)... 
67.207.81.229, 161.35.60.200, 2604:a880:400:d0::1b6b:7001, ...
Connecting to letsencrypt.org (letsencrypt.org)|67.207.81.229|:443... connected.
ERROR: The certificate of ‘letsencrypt.org’ is not trusted.
ERROR: The certificate of ‘letsencrypt.org’ doesn't have a known issuer.


1 Like

If you are asking me... don't purge anything
[that will also remove things that depend on what is being purged]

3 Likes

But at scale with more than a single domain and more than a single server to do all that manual work? Maybe certbot could add a new option --rewrite-chain which can run and just do as the name suggests, replace the chain .pem files with updated changes i.e. --rewrite-chain "ISRG Root X1" and it would do that across all issued certificates on the server ?

3 Likes

It's of course possible to script such a thing using e.g. sed or something similar.

I've asked for a chain switching option without forcibly renewing already: Save alternate chain(s) to disk at issuance · Issue #9052 · certbot/certbot · GitHub

4 Likes

sweet thanks for the link!

1 Like

Is there a conf setting for the --preferred-chain "ISRG Root X1" option, so that certbot on next renewal doesn't re-add the expired root?

I removed it manually, and now DOT on my Samsung S10+ is happy to run through Stunnel again, but it will be annoying if that breaks again in three months.

I only have certbot version 0.40.0 and Linux Mint doesn't have a newer version available.

2 Likes

Note that the --preferred-chain command line option isn't available for certbot 0.40.0: it's way too old. Perhaps you could use other methods of installing certbot. E.g., the currently recommended way is using "snap". Or perhaps using pip. See https://certbot.eff.org/ for more info.

Also, selecting a different chain through the command line without forcibly renewing the certificate is not possible unfortunately. It's possible to edit the renewal configuration file manually though, but you'd need certbot 1.12.0 or newer.

3 Likes

Using snap worked like a charm. Now my phone is happily running DOT through Stunnel.

3 Likes

Note that "just installing certbot using snap" does not actually set the --preferred-chain "ISRG Root X1" option! It's just that a recent version of certbot is required for support for the --preferred-chain command line option.

3 Likes

Yes, but running it with that option did push the option into the conf automatically. I didn't have to add the option manually.

1 Like