Certbot fails to renew a certificate that has expired a few hours before

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: redhawk.concurrent-rt.com

I ran this command: certbot certificates

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
Certificate Name: redhawk.concurrent-rt.com
Domains: redhawk.concurrent-rt.com
Expiry Date: 2021-12-04 10:51:11+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/redhawk.concurrent-rt.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/redhawk.concurrent-rt.com/privkey.pem

I then ran the following command and have included its output:
certbot renew --cert-name redhawk.concurrent-rt.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/redhawk.concurrent-rt.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Attempting to renew cert (redhawk.concurrent-rt.com) from /etc/letsencrypt/renewal/redhawk.concurrent-rt.com.conf produced an unexpected error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/redhawk.concurrent-rt.com/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/redhawk.concurrent-rt.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

My web server is (include version): apache2 2.4.18-2ubuntu3

The operating system my web server runs on is (include version): Ubuntu 1.6.04

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.28.0

The system involved is a customer-facing system and I need to get it back only quickly.
Thanks for any pointers you can give me.

3 Likes
2

Welcome to the community @real-time-jeff

Your CA root certificates are likely out of date. The below post resolved that for other Ubuntu 16 people so will probably work for you too. Post back if that does not help. Or, post back and let us know it did too - either way :slight_smile:

3 Likes
3

I had resolved the root expiry problem back when it expired in September.

The site was working yesterday.

I did not realize the certificate expired today -- auto renewal has been working for me
for years -- don't know why it stopped working.

I did apt install ca-certificates, and it was already up to date.

Not sure what to do next -- to see the real reason that the validity check failed.

From all my reading today, it seems that perhaps once the certificate has expired, I can't renew it???

2 Likes
4

The renew failed due to the SSL: CERTIFICATE_VERIFY_FAILED for certbot trying to reach the Lets Encrypt server.

This is usually due to stale CA cert store. Your last successful cert renewal was early Sept before the DST Root CA X3 expired. At that time the LE server switched to a new chain that ended with ISRG Root X1. Also at that time your cert "renewal" did not have to do anything as your cert was fresh enough. Only recently would certbot have tried to actually renew and needed to use ISRG Root X1. The Certificate Verify failure is almost certainly due to this ISRG Root X1.

What does this command show? (tests simple access to LE server)

curl -I https://acme-v02.api.letsencrypt.org

Also, what does this show? (shows state of key root certs)

grep -E 'ISRG Root|DST Root' /etc/ssl/certs/ca-certificates.crt

Update: Oh, and this is absolutely not true:

it seems that perhaps once the certificate has expired, I can't renew it???

5 Likes
5

The first command returned the following:
redhawk> sudo curl -I https://acme-v02.api.letsencrypt.org
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: curl - SSL CA Certificates

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

The second command did not return any output -- it failed to match the regular expression (return code 1).

The certificate file in complained about his significant content:
redhawk> ls -l /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 274340 Apr 20 2016 /etc/ssl/certs/ca-certificates.crt

But apparently not what it should have. Just noticed that file hasn't been modified for a long time.

I'm afraid I don't know how to proceed at this point. Should I start over completely and try to
wipe out everything wrt certbot/lets-encrypt and get a new certificate? I though there would be
fewer issue for users if I was able to extend/renew the current certificate - clearly out of my depth here.

I examined the certificate via Chrome and it showed the following hierarchy:
Certificate Hierarchy
ISRG Root X1
R3
redhawk.concurrent-rt.com

2 Likes
6

Did you do all these commands when updating CA store?

Do:
sudo apt-get update
sudo apt update

Then, this and show output:
sudo apt install ca-certificates

That has been working for other Ubuntu 16 people. You are definitely missing ISRG Root X1

Also, using a browser to see the chain is not that helpful. Browsers make guesses about the chain to adapt to poorly configured servers and other reasons. It is best to use sites like this one. SSL Checker

UPDATE:
Regarding the below command you should have seen something (DST Root with the date shown for the file). Be sure to enter it with the case shown. And, maybe add one term to ensure you see something:

grep -E 'ISRG Root|DST Root|Daddy' /etc/ssl/certs/ca-certificates.crt

I strongly believe once you have your certs updated the rest will go smooth

3 Likes
7

BINGO!!!

3 Likes
9

Thank you for all your help, my server is back on line with a valid certificate.

Read on for how I did it and the troubles on the way in case this might help someone:

I ran the grep -E command mentioned above exactly the way you requested. No matches.
I then reran it with -i (insensitive) and still got no matches. In fact, I viewed the file and got
no words at all, just ASCII gibberish, the standard kind of thing you see in keys files.

I again tried to update, (sudo apt-get update) but it complains about the the following repository:
deb Index of /certbot/certbot/ubuntu xenial main
which complained that the public key wasn't available and and a few more messages, but none
of them looked fatal:

redhawk> sudo apt-get update
Hit:1 Index of /ubuntu xenial InRelease
Get:2 Index of /certbot/certbot/ubuntu xenial InRelease [24.3 kB]
Ign:2 Index of /certbot/certbot/ubuntu xenial InRelease
Fetched 24.3 kB in 0s (63.5 kB/s)
Reading package lists... Done
W: GPG error: Index of /certbot/certbot/ubuntu xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8C47BE8E75BCA694
W: The repository 'Index of /certbot/certbot/ubuntu xenial InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: There is no public key available for the following key IDs

$ root> apt-get install ca-certificates
Got ~Current version is up to date.

I downloaded the certtool utility and ran it:
certtool -i < /etc/ssl/certs/ca-certificates.crt
and it gave very verbose output, but only Daddy showed up. I got no matches on ISRG Root or DST Root

Finally, I snarfed a /etc/ssl/certs/ca-certificates.crt from an Ubuntu 20.04 system, then renewed (successfully!), restarted apache2, and things worked.

I was nervous about just grabbing that newer ca-certificates.crt file from another system,
but didn't have much choice because, for whatever reason, I could not update it via apt.

Thanks again for all your help.

4 Likes
10

So it's a "catch 22", the update you were looking for was unable to be reached because the system providing the update now requires the update to be in place before it can be reached "securely".

Interesting...
Does sudo apt-get update now run without issue?

3 Likes
11

Nope, getting the same error as before... not signed, no public key available.

3 Likes
12

Not catch 22 - that's good news.

Here is what is going on:

3 Likes
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.