Renewals failing after upgrade to debian 11

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
setpad.ca

I ran this command:
sudo certbot renew --dry-run

It produced this output:

Processing /etc/letsencrypt/renewal/app.setpad.ca.conf

Simulating renewal of an existing certificate for app.setpad.ca

Processing /etc/letsencrypt/renewal/setpad.ca-0001.conf

Simulating renewal of an existing certificate for setpad.ca and 2 more domains

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: setpad.ca
Type: unauthorized
Detail: 141.193.213.11: Invalid response from https://farm-ingress-api.wpesvc.net/v1/pki_validation/BNeq-PGvORgunNL4d5-ttcSP3vh_YsEcj5jyi77IhPo: 404

Domain: www.setpad.ca
Type: unauthorized
Detail: 141.193.213.11: Invalid response from https://farm-ingress-api.wpesvc.net/v1/pki_validation/M0fRVBs7Pu66dcY76-PVRBdKdf9RIMkJWeopdpMZ1j0: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Failed to renew certificate setpad.ca-0001 with error: Some challenges have failed.

Processing /etc/letsencrypt/renewal/setpad.ca.conf

Renewal configuration file /etc/letsencrypt/renewal/setpad.ca.conf is broken.
The error was: renewal config file {} is missing a required file reference
Skipping.

The following simulated renewals succeeded:
/etc/letsencrypt/live/app.setpad.ca/fullchain.pem (success)

The following simulated renewals failed:
/etc/letsencrypt/live/setpad.ca-0001/fullchain.pem (failure)

Additionally, the following renewal configurations were invalid:
/etc/letsencrypt/renewal/setpad.ca.conf (parsefail)

1 renew failure(s), 1 parse failure(s)

My web server is (include version):
nginx version: nginx/1.10.3

The operating system my web server runs on is (include version):
Debian GNU/Linux 11 (bullseye)

My hosting provider, if applicable, is:
Gcloud

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.8.0

*** Also note, several nasty looking errors in /var/log/letsencrypt/letsencrypt.log
2024-01-07 08:49:37,438:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/snap/certbot/3566/bin/certbot", line 8, in
sys.exit(main())
File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/main.py", line 1869, in main
return config.func(config, plugins)
File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/main.py", line 1600, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/main.py", line 143, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/client.py", line 528, in obtain_and_enroll_certificate
return storage.RenewableCert.new_lineage(
File "/snap/certbot/3566/lib/python3.8/site-packages/certbot/_internal/storage.py", line 1084, in new_lineage
raise errors.CertStorageError(
certbot.errors.CertStorageError: archive directory exists for setpad.ca
2024-01-07 08:49:37,443:ERROR:certbot._internal.log:archive directory exists for setpad.ca

2

More context:
This system was upgraded by first cloning the system drive, then proceeding to the debian upgrade, and finally re-attaching the system drive to the host instance.
Although, when I go back to the original system drive image, certbot is also failing to renew, yet this drive has been up and running for several months with no certificate issues.

3

Can you explain why you redirect requests for the setpad.ca domain to farm-ingress-api.wpesvc.net ?

Have you always had your setpad domain proxied at Cloudflare?

Looks like a damaged renewal conf file. Is this a new error?

Or, was this damage the reason you needed to create a -0001 version of it?

4 Likes
4

Thanks for the response.
On your first question: I cannot explain. I don't know what that it. Nothing deliberate on my part. This was installed years ago and hasn't changed recently (aside from yesterday's upgrade).

Second question: The -0001 file has been there for a while but I can't recall why. The important domain and hosts that need to be certified are setpad.ca, app.setpad.ca, and www.setpad.ca

Also, prior to raising this ticket, I did read this medium article:

And after completing all the steps and running this command:
doe@localhost:~$ sudo certbot certonly --manual --preferred-challenges=dns --email --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d *.setpad.ca
I got the error ... "certbot.errors.CertStorageError: archive directory exists for setpad.ca"

But I just re-ran the command (one hour later) and it is now successful

5

Well, that's not very good. You cannot auto-renew a --manual request without appropriate hook. And, since you have an nginx server you should use --webroot or even --nginx method.

You have quite a lot gone wrong here but I do not have enough time to walk you through all the remedy. You should wait for another volunteer to pick up or maybe I will later today.

3 Likes
6

Ok. Thanks again for your help. So If I am not mistaken, at least the certificates are up and running, just not auto-renewing yes? So I can wait to continue when you can.

1 Like
7

I am not sure where to begin here. I see you got a wildcard cert with the --manual command but I don't see that you are using it anywhere.

Requests to your setpad.ca and its www subdomain use a cert from Cloudflare Inc. Is this domain handled by a service called "WP Engine"?

Requests to your app.setpad.ca use a Let's Encrypt cert issued on Nov18 with just that name in it. This app subdomain has a different public IP than your other domain.

4 Likes
8

Apologies for the confusion. I used the wildcard cert because that's what was given in the example of the medium article. And your questions are jogging my memory.

setpad.ca and its www are now indeed on Wordpress hosting and I recall that their certs are managed there.

app.setpad.ca is the host that was just upgraded, running nginx, and from which I am currently running the commands and diagnostics posted in this ticket.

3 Likes
10

May we see the output of?:

certbot certificates

And your name put a smile on my face :slight_smile:

2 Likes
11

I will deny any reference to Chris Farley. Can't handle those images him in a Chippendales outfit... :see_no_evil:

cat /etc/letsencrypt/renewal/app.setpad.ca.conf

# renew_before_expiry = 30 days
version = 2.7.4
archive_dir = /etc/letsencrypt/archive/app.setpad.ca
cert = /etc/letsencrypt/live/app.setpad.ca/cert.pem
privkey = /etc/letsencrypt/live/app.setpad.ca/privkey.pem
chain = /etc/letsencrypt/live/app.setpad.ca/chain.pem
fullchain = /etc/letsencrypt/live/app.setpad.ca/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 13bab1c203baccd26c8a12fdecc82ca4
authenticator = nginx
installer = nginx
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa
post_hook = systemctl reload nginx

cat /etc/letsencrypt/renewal/setpad.ca-0001.conf

# renew_before_expiry = 30 days
version = 1.23.0
archive_dir = /etc/letsencrypt/archive/setpad.ca-0001
cert = /etc/letsencrypt/live/setpad.ca-0001/cert.pem
privkey = /etc/letsencrypt/live/setpad.ca-0001/privkey.pem
chain = /etc/letsencrypt/live/setpad.ca-0001/chain.pem
fullchain = /etc/letsencrypt/live/setpad.ca-0001/fullchain.pem

# Options used in the renewal process
[renewalparams]
installer = nginx
server = https://acme-v02.api.letsencrypt.org/directory
account = 13bab1c203baccd26c8a12fdecc82ca4
authenticator = nginx

cat /etc/letsencrypt/renewal/setpad.ca.conf:

- nothing -

certbot certificates

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/setpad.ca.conf produced an unexpected error: renewal config file {} is missing a required file reference. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: app.setpad.ca
    Serial Number: 3331feeef637eed228efcebd78e3267e4ec
    Key Type: RSA
    Domains: app.setpad.ca
    Expiry Date: 2024-02-16 11:19:57+00:00 (VALID: 39 days)
    Certificate Path: /etc/letsencrypt/live/app.setpad.ca/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/app.setpad.ca/privkey.pem
  Certificate Name: setpad.ca-0001
    Serial Number: 4db244ddbb5c414cfdb1b074289b4c4502c
    Key Type: RSA
    Domains: setpad.ca app.setpad.ca www.setpad.ca
    Expiry Date: 2022-05-25 05:21:31+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/setpad.ca-0001/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/setpad.ca-0001/privkey.pem
  Certificate Name: setpad.ca-0002
    Serial Number: 3ed0097876e812178f64f8643895da95efe
    Key Type: ECDSA
    Domains: *.setpad.ca
    Expiry Date: 2024-04-06 13:48:09+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/setpad.ca-0002/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/setpad.ca-0002/privkey.pem

The following renewal configurations were invalid:
  /etc/letsencrypt/renewal/setpad.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
12

Seems like the first cert is included in the second cert - so, that first one can be removed.

The third cert is helpful for other/future names [but doesn't cover the base domain "setpad.ca"] - so, it can't fully replace the second cert.

This is weird/unexpected:

I suppose at some point you may have manually deleted some file(s) within the /etc/letsencrypt folder.
It should be "cleaned up" / deleted:
rm /etc/letsencrypt/renewal/setpad.ca.conf

2 Likes
13

Make sure you point the web server to a remaining cert BEFORE you delete any cert(s).

1 Like
14

Okay. Well, this is getting easier to sort out now.

I see @rg305 and I were typing at the same time. But, we can delete the unused certs. Your WP Engine service is handling those.

We only keep the one for app.setpad.ca in it.

And, we will delete that empty conf file (rg305 already showed that command)

We will then test the renew for app to ensure it will renew when it is due.

So, go ahead and issue these

certbot delete --cert-name setpad.ca-0001
certbot delete --cert-name setpad.ca-0002
2 Likes
15

but I liked 0002, it was the only one that was ECDSA...
LOL

3 Likes
16

and, was acquired manually just because google said so ... :slight_smile:

3 Likes
17

I completely overlooked the

value with -0001.
That makes my statement incorrect:

The second cert is invalid :frowning:

3 Likes
18

It was a

provided by "Dr. Google"
[that's the new name - indelibly etched into my eyelids - LOL]

2 Likes
19

Ok. Recapping before I proceed. Please confirm,

rm /etc/letsencrypt/renewal/setpad.ca.conf
certbot delete --cert-name setpad.ca-0001
certbot delete --cert-name setpad.ca-0002

1 Like
20

Make sure the web service isn't using any of those.
It shouldn't be - but it pays to measure twice and cut once.

3 Likes
21

grep cert /etc/nginx/sites-enabled/setpad.ca
ssl_certificate /etc/letsencrypt/live/app.setpad.ca/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/app.setpad.ca/privkey.pem; # managed by Certbot
proxy_set_header X-SSL-CERT $ssl_client_cert; # attention: Deprecated in recent versions. replaced by $ssl_client_escapte_cert
proxy_set_header X-SSL-CERT $ssl_client_cert; # attention: Deprecated in recent versions. replaced by $ssl_client_escapte_cer

1 Like