Cant Renew Cert on Some Domains

My certs were set to auto renew using cron 3 of the certs passed while 2 failed and I'm not sure why I'm getting this error below. Are the symlinks broken? See below:

Output of live directory:

drwxr-xr-x 2 root root 4.0K Jan 20 22:41 .
drwx------ 6 root root 4.0K Jan 20 21:21 ..
-rw-r--r-- 1 root root 692 Oct 25 02:43 README
lrwxrwxrwx 1 root root 49 Oct 25 16:25 cert.pem -> ../../archive/jira.sunriselabs.com-0001/cert2.pem
lrwxrwxrwx 1 root root 50 Oct 25 16:25 chain.pem -> ../../archive/jira.sunriselabs.com-0001/chain2.pem
lrwxrwxrwx 1 root root 54 Oct 25 16:25 fullchain.pem -> ../../archive/jira.sunriselabs.com-0001/fullchain2.pem
lrwxrwxrwx 1 root root 52 Oct 25 16:25 privkey.pem -> ../../archive/jira.sunriselabs.com-0001/privkey2.pem

Output of archive directory:

drwxr-xr-x 2 root root 4.0K Oct 25 16:25 .
drwx------ 10 root root 4.0K Oct 25 17:38 ..
-rw-r--r-- 1 root root 1.9K Oct 25 02:43 cert1.pem
-rw-r--r-- 1 root root 1.9K Oct 25 16:25 cert2.pem
-rw-r--r-- 1 root root 1.7K Oct 25 02:43 chain1.pem
-rw-r--r-- 1 root root 1.7K Oct 25 16:25 chain2.pem
-rw-r--r-- 1 root root 3.5K Oct 25 02:43 fullchain1.pem
-rw-r--r-- 1 root root 3.5K Oct 25 16:25 fullchain2.pem
-rw------- 1 root root 1.7K Oct 25 02:43 privkey1.pem
-rw------- 1 root root 1.7K Oct 25 16:25 privkey2.pem

My domain is:

I ran this command:

docker run --rm -v certbotConf:/etc/letsencrypt -v certbotWww:/var/www/certbot -v certbotLib:/var/lib/letsencrypt -v certbotLog:/var/log/letsencrypt certbot/certbot renew >> /var/log/ssl-renew.log 2>&

It produced this output:

Processing /etc/letsencrypt/renewal/jira.sunriselabs.com.conf


Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Attempting to renew cert (jira.sunriselabs.com) from /etc/letsencrypt/renewal/jira.sunriselabs.com.conf produced an unexpected error: [Errno 17] File exists: '/etc/letsencrypt/archive/jira.sunriselabs.com/privkey3.pem'. Skipping

My web server is (include version):
Nginx

The operating system my web server runs on is (include version):
Ubuntu 18.04

My hosting provider, if applicable, is:
AWS

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

Hi and welcome to the LE community forum :slight_smile:

Please show the outputs of:
certbot --version
certbot certificates
ls -l /etc/letsencrypt/archive/jira.sunriselabs.com/
cat /etc/letsencrypt/renewal/jira.sunriselabs.com.conf
ls -l /etc/letsencrypt/archive/jira.sunriselabs.com-0001/
cat /etc/letsencrypt/renewal/jira.sunriselabs.com-0001.conf

Thanks! Sorry I took over for someone that left our company and my cert knowledge isn't the best :frowning:

cerbot version: certbot 1.8.0

certificates:


Found the following certs:
Certificate Name: confluence.sunriselabs.com
Serial Number: 3cd3b5714decc20242ef212eb86b967bbe3
Domains: confluence.sunriselabs.com
Expiry Date: 2021-04-20 19:46:21+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/confluence.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/confluence.sunriselabs.com/privkey.pem
Certificate Name: git.sunriselabs.com
Serial Number: 3192b082c468684fc81c849a2ed73b09e09
Domains: git.sunriselabs.com
Expiry Date: 2021-01-23 16:38:29+00:00 (VALID: 2 days)
Certificate Path: /etc/letsencrypt/live/git.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/git.sunriselabs.com/privkey.pem
Certificate Name: jira.sunriselabs.com
Serial Number: 4a7a1097c07b13837521ab99f348e3397ab
Domains: jira.sunriselabs.com
Expiry Date: 2021-01-23 15:25:14+00:00 (VALID: 2 days)
Certificate Path: /etc/letsencrypt/live/jira.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/jira.sunriselabs.com/privkey.pem
Certificate Name: pgadmin.sunriselabs.com
Serial Number: 401cf592c747fbc1223ec3467c5a73b0ce8
Domains: pgadmin.sunriselabs.com
Expiry Date: 2021-03-23 23:07:42+00:00 (VALID: 61 days)
Certificate Path: /etc/letsencrypt/live/pgadmin.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/pgadmin.sunriselabs.com/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/confluence.sunriselabs.com-0001.conf
/etc/letsencrypt/renewal/git.sunriselabs.com-0001.conf
/etc/letsencrypt/renewal/git.sunriselabs.com-0002.conf
/etc/letsencrypt/renewal/jira.sunriselabs.com-0001.conf


ls -l /etc/letsencrypt/archive/jira.sunriselabs.com/

total 20
-rw-r--r-- 1 root root 1923 Oct 24 22:31 cert1.pem
-rw-r--r-- 1 root root 1647 Oct 24 22:31 chain1.pem
-rw-r--r-- 1 root root 3570 Oct 24 22:31 fullchain1.pem
-rw------- 1 root root 1704 Oct 24 22:31 privkey1.pem
-rw------- 1 root root 1704 Dec 25 00:08 privkey3.pem

cat /etc/letsencrypt/renewal/jira.sunriselabs.com.conf

renew_before_expiry = 30 days

version = 1.8.0
archive_dir = /etc/letsencrypt/archive/jira.sunriselabs.com
cert = /etc/letsencrypt/live/jira.sunriselabs.com/cert.pem
privkey = /etc/letsencrypt/live/jira.sunriselabs.com/privkey.pem
chain = /etc/letsencrypt/live/jira.sunriselabs.com/chain.pem
fullchain = /etc/letsencrypt/live/jira.sunriselabs.com/fullchain.pem

Options used in the renewal process

[renewalparams]
account = 084c927a0555b1698bb875a2d391508b
authenticator = webroot
webroot_path = /var/www/certbot,
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
jira.sunriselabs.com = /var/www/certbot

ls -l /etc/letsencrypt/archive/jira.sunriselabs.com-0001/

total 32
-rw-r--r-- 1 root root 1923 Oct 25 02:43 cert1.pem
-rw-r--r-- 1 root root 1927 Oct 25 16:25 cert2.pem
-rw-r--r-- 1 root root 1647 Oct 25 02:43 chain1.pem
-rw-r--r-- 1 root root 1647 Oct 25 16:25 chain2.pem
-rw-r--r-- 1 root root 3570 Oct 25 02:43 fullchain1.pem
-rw-r--r-- 1 root root 3574 Oct 25 16:25 fullchain2.pem
-rw------- 1 root root 1704 Oct 25 02:43 privkey1.pem
-rw------- 1 root root 1704 Oct 25 16:25 privkey2.pem

cat /etc/letsencrypt/renewal/jira.sunriselabs.com-0001.conf

renew_before_expiry = 30 days

version = 1.8.0
archive_dir = /etc/letsencrypt/archive/jira.sunriselabs.com-0001
cert = /etc/letsencrypt/live/jira.sunriselabs.com-0001/cert.pem
privkey = /etc/letsencrypt/live/jira.sunriselabs.com-0001/privkey.pem
chain = /etc/letsencrypt/live/jira.sunriselabs.com-0001/chain.pem
fullchain = /etc/letsencrypt/live/jira.sunriselabs.com-0001/fullchain.pem

Options used in the renewal process

[renewalparams]
account = 084c927a0555b1698bb875a2d391508b
authenticator = webroot
webroot_path = /var/www/certbot,
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]

This doesn't look right:
[File privkey3.pem seems to be out-of-place (and time)]

Some major issues here:
[seems like system suffered some catastrophic event - ~Oct 24/25]

Let's see what (if anything) remains of the -000 types:
ls -lR /etc/letsencrypt/*-000*

The confluence domain works but git.* and jira.* are failing:

/var/lib/docker/volumes/certbotConf/_data/archive# ls -lR -000

confluence.sunriselabs.com-0001:
total 16
-rw-r--r-- 1 root root 1944 Oct 25 02:30 cert1.pem
-rw-r--r-- 1 root root 1647 Oct 25 02:30 chain1.pem
-rw-r--r-- 1 root root 3591 Oct 25 02:30 fullchain1.pem
-rw------- 1 root root 1708 Oct 25 02:30 privkey1.pem

git.sunriselabs.com-0001:
total 16
-rw-r--r-- 1 root root 1923 Oct 25 16:26 cert1.pem
-rw-r--r-- 1 root root 1647 Oct 25 16:26 chain1.pem
-rw-r--r-- 1 root root 3570 Oct 25 16:26 fullchain1.pem
-rw------- 1 root root 1704 Oct 25 16:26 privkey1.pem

git.sunriselabs.com-0002:
total 16
-rw-r--r-- 1 root root 1923 Oct 25 17:38 cert1.pem
-rw-r--r-- 1 root root 1647 Oct 25 17:38 chain1.pem
-rw-r--r-- 1 root root 3570 Oct 25 17:38 fullchain1.pem
-rw------- 1 root root 1704 Oct 25 17:38 privkey1.pem

jira.sunriselabs.com-0001:
total 32
-rw-r--r-- 1 root root 1923 Oct 25 02:43 cert1.pem
-rw-r--r-- 1 root root 1927 Oct 25 16:25 cert2.pem
-rw-r--r-- 1 root root 1647 Oct 25 02:43 chain1.pem
-rw-r--r-- 1 root root 1647 Oct 25 16:25 chain2.pem
-rw-r--r-- 1 root root 3570 Oct 25 02:43 fullchain1.pem
-rw-r--r-- 1 root root 3574 Oct 25 16:25 fullchain2.pem
-rw------- 1 root root 1704 Oct 25 02:43 privkey1.pem
-rw------- 1 root root 1704 Oct 25 16:25 privkey2.pem

hmm...
This needs some major cleanup.
You can probably delete all of those; as there are no live symlinks, nor functional renewal configs, using any of them.

Then try:
certbot update_symlinks
certbot certificates

Also, check through your nginx config to ensure no -000 files are being used there.
nginx -T | grep '-000'

From the nginx conf file looks like there isn't any reference to 000:

Configuration for Jira - client_max_body_size must be at least the max allowed attachment size

server {
listen 80;
server_name jira.sunriselabs.com;
location / {
return 301 https://$host$request_uri;
}
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
}
server {
listen 443 ssl;
server_name jira.sunriselabs.com;
proxy_read_timeout 600s;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://jira:8080;
client_max_body_size 50M;
}
ssl_certificate /etc/letsencrypt/live/jira.sunriselabs.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/jira.sunriselabs.com/privkey.pem;
include /etc/letsencrypt/conf/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/conf/ssl-dhparams.pem;
}

After updating the symlinks:


Found the following certs:
Certificate Name: confluence.sunriselabs.com
Serial Number: 3cd3b5714decc20242ef212eb86b967bbe3
Domains: confluence.sunriselabs.com
Expiry Date: 2021-04-20 19:46:21+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/confluence.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/confluence.sunriselabs.com/privkey.pem
Certificate Name: pgadmin.sunriselabs.com
Serial Number: 401cf592c747fbc1223ec3467c5a73b0ce8
Domains: pgadmin.sunriselabs.com
Expiry Date: 2021-03-23 23:07:42+00:00 (VALID: 61 days)
Certificate Path: /etc/letsencrypt/live/pgadmin.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/pgadmin.sunriselabs.com/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/confluence.sunriselabs.com-0001.conf
/etc/letsencrypt/renewal/git.sunriselabs.com-0001.conf
/etc/letsencrypt/renewal/git.sunriselabs.com-0002.conf
/etc/letsencrypt/renewal/git.sunriselabs.com.conf
/etc/letsencrypt/renewal/jira.sunriselabs.com-0001.conf
/etc/letsencrypt/renewal/jira.sunriselabs.com.conf


The symlink is now broken for jira.sunriselabs.com renewal:

/var/lib/docker/volumes/certbotConf/_data/live/jira.sunriselabs.com# ls -alhR
.:
total 12K
drwxr-xr-x 2 root root 4.0K Jan 20 22:41 .
drwx------ 6 root root 4.0K Jan 20 21:21 ..
-rw-r--r-- 1 root root 692 Oct 25 02:43 README
lrwxrwxrwx 1 root root 49 Oct 25 16:25 cert.pem -> ../../archive/jira.sunriselabs.com-0001/cert2.pem
lrwxrwxrwx 1 root root 50 Oct 25 16:25 chain.pem -> ../../archive/jira.sunriselabs.com-0001/chain2.pem
lrwxrwxrwx 1 root root 54 Oct 25 16:25 fullchain.pem -> ../../archive/jira.sunriselabs.com-0001/fullchain2.pem
lrwxrwxrwx 1 root root 52 Oct 25 16:25 privkey.pem -> ../../archive/jira.sunriselabs.com-0001/privkey2.pem

You still have broken renewals; some with -000s

It may be simplest to just issue a new jira.sunriselabs.com cert.
Be careful here; as three certs have already been issued for that name within the past seven days:
crt.sh | jira.sunriselabs.com

Do I need to symlink the new certs manually? Since they're still broken?

No.
You do need for certbot to think is has a cert in order to renew though.
Right now it only think it has these certs:

Otherwise it becomes like a first time issuance.

Oh, it looks like I made that mistake. I ran a new cert, how do I make certbot think it has a cert to renew?

Did it get a new cert?
If so, we can use that.

Renewals are automatically based on known configs.
Which are based on the last issuance.
If none is known, then a new renewal config is created.

/var/lib/docker/volumes/certbotConf/_data/live/jira.sunriselabs.com# docker run --rm -it \

-v certbotConf:/etc/letsencrypt
-v certbotWww:/var/www/certbot
-v certbotLib:/var/lib/letsencrypt
-v certbotLog:/var/log/letsencrypt
certbot/certbot
certonly --webroot -w /var/www/certbot -d jira.sunriselabs.com --force-renewal --agree-tos
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate

IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/jira.sunriselabs.com-0002/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/jira.sunriselabs.com-0002/privkey.pem
    Your cert will expire on 2021-04-21. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot
    again. To non-interactively renew all of your certificates, run
    "certbot renew"

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let's Encrypt: Donate - Let's Encrypt
    Donating to EFF: Support EFF's Work on Let's Encrypt | Electronic Frontier Foundation

Let's see if all the gears turned properly....

certbot certificates


Found the following certs:
Certificate Name: confluence.sunriselabs.com
Serial Number: 3cd3b5714decc20242ef212eb86b967bbe3
Domains: confluence.sunriselabs.com
Expiry Date: 2021-04-20 19:46:21+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/confluence.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/confluence.sunriselabs.com/privkey.pem
Certificate Name: jira.sunriselabs.com-0002
Serial Number: 3b2c6bdfac1f4b5818331295f7ad856425d
Domains: jira.sunriselabs.com
Expiry Date: 2021-04-21 05:27:53+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/jira.sunriselabs.com-0002/fullchain.pem
Private Key Path: /etc/letsencrypt/live/jira.sunriselabs.com-0002/privkey.pem
Certificate Name: pgadmin.sunriselabs.com
Serial Number: 401cf592c747fbc1223ec3467c5a73b0ce8
Domains: pgadmin.sunriselabs.com
Expiry Date: 2021-03-23 23:07:42+00:00 (VALID: 61 days)
Certificate Path: /etc/letsencrypt/live/pgadmin.sunriselabs.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/pgadmin.sunriselabs.com/privkey.pem

Not the tidiest, but useable:

Now be sure your nginx config uses the path (exactly as) shown :

[otherwise they will get out-of-sync]

Actually there seems to be one cert missing from that list:
git.sunriselabs.com

Yah, I didn't issue another cert for it. I was hoping to learn how to renew it instead issuing a new cert.