Found bug in my renewal script and have a queston

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mail.jewettfarm.com, mail.jewettracing.com, mail.mail-lab.us

I ran this command: certbot renew

It produced this output: ERROR:certbot._internal.renewal:Failed to renew
certificate mail.jewettfarm.com with error: [Errno 2]
No such file or directory: '/etc/letsencrypt/archive/mail.jewettfarm.com/privkey6.pem'

My web server is (include version): Nginx 1.18

The operating system my web server runs on is (include version): Ubuntu 20.04

My hosting provider, if applicable, is: Cloudflare for DNS-001 validation using plugin

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Certbot 2.5

I see my script was unable to locate a directory to save the privkey.pem file. I have corrected this. This got missed during a container migration process last week and I screwed up.

My question, can I use the last acme response "-----BEGIN CERTIFICATE----" information I found in /var/log/letsencrypt/letsencrypt.log, since I am rate limited for 24 hours. Or am I stuck waiting?

Thank you

You can retreive the certificate(s) from the logs indeed. Or use a service such as https://crt.sh/ although they are often overloaded and backlogged.

Also, why would you be rate limited?

Holy cow, looking at your certificate issuance (https://crt.sh/?deduplicate=Y&q=mail.mail-lab.us), what are you DOING? Looks like you're getting a certificate almost DAILY? Why?

3 Likes

No, the privkey.pem is not included in the letsencrypt.log file

You should be able to get a fresh cert tomorrow once your Apr8 cert "rolls off". But, you should quit getting a cert every day or so. Your cert history is here:
https://tools.letsdebug.net/cert-search?m=domain&q=mail.jewettfarm.com&d=744

4 Likes

Sorry, I forgot to add that info:

It kept trying with the directory error that I did not catch until this morning. Max of 5 attempts was reached.

Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: mail.jewettfarm.com,mail.jewettracing.com,mail.mail-lab.us, retry after 2023-04-15T07:12:49Z: see Duplicate Certificate Limit - Let's Encrypt

So in the logs I see two certificate strings, is the first the certificate, and second "----BEGIN CERTIFICATE----" block is the privkey file? Just need to know how to match them up. I can just paste them into new files in the correct path.
Thanks!

1 Like

No, that's a certificate, not a key. Keys are not logged. The second (and possibly third) certificates are the certificate chain.

Also, as already said (in case you missed it above), please stop getting certs (almost) DAILY. Why are you doing that?

4 Likes

No, you actually got 5 certs in last week and that's the limit. See below.

As I noted, the privkey is not in the log

But please start using persistent storage to save your cert files. You have numerous un-expired certs outstanding. It's more than 20 certs just in the past 30 days

4 Likes

Yes, I am digging into why on the daily renewals, they are set to cron monthly at the most. So something is off. Killing the cron jobs for now till I get this sorted out.

You should not do just monthly renewal attempts. If it fails due to comms or LE issue you won't try again until the next month which might be too late.

The recommendation is to use certbot renew daily. If the cert is not due for renewal nothing is done.

You possibly have used --force-renewal or similar which often causes grief

4 Likes

Usually this is due to the usage of the --force-renewal option.

3 Likes

No, the main issue is the directory where the certs are stored was moved, and I missed this in my review of the changes.

For clarity, here are the lines that run from the cron job script:

/usr/bin/certbot certonly
-d mail.jewettfarm.com
-d mail.mail-lab.us
-d mail.jewettracing.com
--manual-public-ip-logging-ok -n
--preferred-chain "ISRG Root X1"
--agree-tos
--register-unsafely-without-email
--dns-cloudflare
--dns-cloudflare-credentials ~/.secrets/certbot/cloudflare.ini

I have moved the /etc/letsencrypt/archive/mail.jewettracing.com directory to it's proper location. Somehow it was renamed as "mail.jewettfarm.com" and the initial certs were pulled in under mail.jewettracing.com - so the renewal had no place to write. I am fixing these issues, just wanted to know if I can somehow manually update the certs with one of the many it has already been sent.

Sorry for the confusion. I need to add my email to the command string so I get these notifications sooner.

Here you say the migration was "last week", but your "daily" certificate issuance is going on since 2023-03-13 already? That's not 1 week earlier, but 1 month? So I'm not fully convinced. Or multiple things are going wrong, starting 2023-03-13, that's possible too of course.

Certs? Sure. Private keys? Nope, not with Certbot 2.5. With Certbot 2.2.0 or earlier, the private keys were saved in /keys/, but you're using 2.5 so no private keys were saved except for in /archive/.

4 Likes

and that is the directory that was moved - so it seems to have been stuck in a loop somehow since it could not write to the archive path.

This may be when the directories were renamed initially, breaking the script process. So a multitude of issues.

Sorry if this is a duplicate, but this is the event that seems to have started this mess back in March:

ERROR:certbot._internal.renewal:Failed to renew certificate mail.jewettfarm.com with error: [Errno 2] No such file or directory: '/etc/letsencrypt/archive/mail.jewettfarm.com/privkey6.pem'
FileNotFoundError: [Errno 2] No such file or directory: '/etc/letsencrypt/archive/mail.jewettfarm.com/privkey6.pem'

Still don't know what it was running daily yet, but this seems to be the trigger point.

Running twice a day is recommended currently, so that's fine. But usually Certbot is able to store the issued cert, except when that's not possible, e.g. due to your migration issue.

With ARI it might be recommended to run your ACME client even more frequent!

It's probably a good idea to have your cronjob setup so it'll notify the responsible person(s) when a cronjob fails when it shouldn't.

4 Likes

Thanks for the help guys. This has been a learning experience for sure, never move certificates path!!

Yes, I am an idiot. lol

2 Likes

Usually one just transfers the entire /etc/letsencrypt/ directory using e.g. rsync with the --archive option or tar. It's also perfectly possible to use e.g. cp but you'd want to use the --archive option for that one too.

5 Likes

So I thought I would try the workaround in Duplicate Certificate Limit - Let's Encrypt (letsencrypt.org)

It renewed the cert for mail.jewettfarm.com, however the private key does not match the certificate in the new directory it created. Oh and it saved the fullchain.pem file as a file and not a symlink, the symlink is named fullchain.pem.orig. So im stumped. Guess I have to wait.

What does this show?

certbot certificates
3 Likes

It seems as though all of that could be replaced with a simple certbot renew. Why are you doing it this way?

4 Likes