Failed renewal... cert files are empty

Help
1

Hi,

I've provided the required details of my setup below.

The letsencrypt cert on an openvpn server we manage expired today.

We ran the command certbot certonly --dns-route53 -d as.neontv.co.nz --post-hook /usr/local/openvpn_as/scripts/update_cert hoping to renew the certificate.

The command failed with the logs below.

I can see the current certificate now looks like...

ls -ltr /etc/letsencrypt/live/as.neontv.co.nz

total 4
-rw-r--r-- 1 root root 692 Feb 21 13:14 README
lrwxrwxrwx 1 root root 42 Feb 27 14:35 privkey.pem -> ../../archive/as.neontv.co.nz/privkey5.pem
lrwxrwxrwx 1 root root 44 Feb 27 14:35 fullchain.pem -> ../../archive/as.neontv.co.nz/fullchain5.pem
lrwxrwxrwx 1 root root 40 Feb 27 14:35 chain.pem -> ../../archive/as.neontv.co.nz/chain5.pem
lrwxrwxrwx 1 root root 39 Feb 27 14:35 cert.pem -> ../../archive/as.neontv.co.nz/cert5.pem

I can also see that the files pointed to under archive are empty...

ls -ltr /etc/letsencrypt/archive/as.neontv.co.nz

total 64
-rw------- 1 root root 241 Feb 21 13:14 privkey1.pem
-rw-r--r-- 1 root root 3315 Feb 21 13:14 fullchain1.pem
-rw-r--r-- 1 root root 1826 Feb 21 13:14 chain1.pem
-rw-r--r-- 1 root root 1489 Feb 21 13:14 cert1.pem
-rw------- 1 root root 241 Feb 27 14:18 privkey2.pem
-rw-r--r-- 1 root root 3319 Feb 27 14:18 fullchain2.pem
-rw-r--r-- 1 root root 1826 Feb 27 14:18 chain2.pem
-rw-r--r-- 1 root root 1493 Feb 27 14:18 cert2.pem
-rw------- 1 root root 241 Feb 27 14:21 privkey3.pem
-rw-r--r-- 1 root root 3315 Feb 27 14:21 fullchain3.pem
-rw-r--r-- 1 root root 1826 Feb 27 14:21 chain3.pem
-rw-r--r-- 1 root root 1489 Feb 27 14:21 cert3.pem
-rw------- 1 root root 241 Feb 27 14:25 privkey4.pem
-rw-r--r-- 1 root root 3319 Feb 27 14:25 fullchain4.pem
-rw-r--r-- 1 root root 1826 Feb 27 14:25 chain4.pem
-rw-r--r-- 1 root root 1493 Feb 27 14:25 cert4.pem
-rw-r--r-- 1 root root 0 Feb 28 15:18 fullchain5.pem
-rw-r--r-- 1 root root 0 Feb 28 15:18 chain5.pem
-rw-r--r-- 1 root root 0 Feb 28 15:18 cert5.pem
-rw------- 1 root root 0 Feb 28 15:18 privkey5.pem

So, I can understand the error in the logs.

But I don't understand why the files are empty.

If I run systemctl list-timers, I can see...
Tue 2024-05-28 02:18:00 NZST 10h left Mon 2024-05-27 13:13:12 NZST 2h 58min ago snap.certbot.renew.timer snap.certbot.renew.service

Config file for letsencrypt renewal looks like this...

cat /etc/letsencrypt/renewal/as.neontv.co.nz.conf

version = 2.9.0
archive_dir = /etc/letsencrypt/archive/as.neontv.co.nz
cert = /etc/letsencrypt/live/as.neontv.co.nz/cert.pem
privkey = /etc/letsencrypt/live/as.neontv.co.nz/privkey.pem
chain = /etc/letsencrypt/live/as.neontv.co.nz/chain.pem
fullchain = /etc/letsencrypt/live/as.neontv.co.nz/fullchain.pem

[renewalparams]
account = ouraccountnumber
authenticator = dns-route53
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
post_hook = /usr/local/openvpn_as/scripts/update_cert

Basic details...

My domain is:
as.neontv.co.nz

I ran this command:
certbot certonly --dns-route53 -d as.neontv.co.nz --post-hook /usr/local/openvpn_as/scripts/update_cert

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
verifying the signature of the certificate located at /etc/letsencrypt/live/as.neontv.co.nz/cert.pem has failed. Details: Unable to load PEM file. See Frequently asked questions — Cryptography 43.0.0.dev1 documentation for more details. MalformedFraming
Traceback (most recent call last):
*** File "/snap/certbot/3700/lib/python3.8/site-packages/certbot/crypto_util.py", line 303, in verify_renewable_cert_sig***
*** chain = x509.load_pem_x509_certificate(chain_file.read(), default_backend())***
ValueError: Unable to load PEM file. See Frequently asked questions — Cryptography 43.0.0.dev1 documentation for more details. MalformedFraming
Renewal configuration file /etc/letsencrypt/renewal/as.neontv.co.nz.conf produced an unexpected error: verifying the signature of the certificate located at /etc/letsencrypt/live/as.neontv.co.nz/cert.pem hasfailed. Details: Unable to load PEM file. See Frequently asked questions — Cryptography 43.0.0.dev1 documentation for more details. MalformedFraming. Skipping.

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/as.neontv.co.nz.conf

My web server is (include version):
openvpn

The operating system my web server runs on is (include version):
ubuntu 22.04

I can login to a root shell on my machine (yes or no, or I don't know):
yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.10.0

2

Maybe try with sudo?

4 Likes
3

Hi,

Thank you for the quick reply.

We did run it as root first.

And then tried using sudo to ensure AWS credentials were getting set.
Which they were...
Found credentials from IAM Role: neon-prod-openvpn-as-role

2 Likes
4 
PORT    STATE    SERVICE
22/tcp  filtered ssh
80/tcp  filtered http
443/tcp open     https

I know you are using a DNS challenge, but it is still best practice to open port 80.

2 Likes
5

Hi @philenz,

You could try more debugging output (i.e. certbot -vv) and look at the logs.
By default Certbot stores status logs in /var/log/letsencrypt.
See User Guide — Certbot 2.10.0 documentation for details about where the log files are.

Here is documentation for Welcome to certbot-dns-route53’s documentation! — certbot-dns-route53 0 documentation

1 Like
6

Hi all,

Thanks for your replies on this one.

We've now figured out what was wrong.

It looks like our timer job failed to do the upgrade and left a bunch of zero length cert files behind in the archive directory.

I replaced them with the previous version and ran "certbot -vv certonly --dns-route53 -d as.neontv.co.nz --post-hook /usr/local/openvpn_as/scripts/update_cert"

And this has worked perfectly.

We just need to take a look at the timer job (snap.certbot.renew.timer) and figure out why it failed.

5 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.