Having issues running CERTBOT Docker Container via CRONJOB but fine when ran through terminal

Help
1

I'm having difficulties running certbot renewals via cronjob in one particular environment. I have two other environments that the cronjob renewals run fine. Only in this one have I received problems. If I set the cronjob a few minutes or hours ahead, it typically runs fine. If I set it to run a week ahead, I run into the issue I describe below.

My domain is: dev.elselabs.io

I ran this command:
I run a CRONTAB which runs this script:

echo "\n BEGINNING REFRESH \n";

# Print the current date and time
dt=$(date '+%d/%m/%Y %H:%M:%S');
echo "$dt"

# Request a Lets Encrypt certificate using the offical Docker image
# for provided domain

DOMAIN=$1;
echo "\n\033[0;33m=> Requesting CERT for $DOMAIN ------------------------------\033[0";
docker run --name certbot -v /etc/letsencrypt:/etc/letsencrypt -v /var/www/html:/var/www/html certbot/certbot certonly \
	   --webroot --webroot-path=/var/www/html -d $DOMAIN --force-renewal --verbose

# GET Nginx docker container name, then stop and remove it
# The automatically generated Nginx container will apply the new CERT

CONTAINER_NAME=$(exec docker ps --format "{{.Names}}" | grep nginx);
echo "\n\033[0;33m=> Stopping $CONTAINER_NAME  ------------------------------\033[0m";
docker stop $CONTAINER_NAME
echo "\n\033[0;33m=> Removing $CONTAINER_NAME  ------------------------------\033[0m";
docker rm $CONTAINER_NAME;

echo "\n REFRESH COMPLETE \n";

The CRONTAB looks like this:

0 0 * * FRI /usr/src/nginx/refresh_cert.sh dev.elselabs.io >> /usr/src/nginx/refresh_cert.err

It produced this output:
When the command is run from the terminal, it looks like this:

\n BEGINNING REFRESH \n
02/02/2024 15:19:07
\n\033[0;33m=> Requesting CERT for dev.elselabs.io ------------------------------\033[0
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate for dev.elselabs.io

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/dev.elselabs.io/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/dev.elselabs.io/privkey.pem
This certificate expires on 2024-05-02.
These files will be updated when the certificate renews.
NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
\n\033[0;33m=> Stopping development_nginx.1.zq9ajwgosjmzxwp9if1xkswqn  ------------------------------\033[0m
development_nginx.1.zq9ajwgosjmzxwp9if1xkswqn
\n\033[0;33m=> Removing development_nginx.1.zq9ajwgosjmzxwp9if1xkswqn  ------------------------------\033[0m
development_nginx.1.zq9ajwgosjmzxwp9if1xkswqn
\n REFRESH COMPLETE \n

When the crontab runs it, it looks like this:

\n BEGINNING REFRESH \n
02/02/2024 00:00:01
\n\033[0;33m=> Requesting CERT for dev.elselabs.io ------------------------------\033[0
Renewing an existing certificate for dev.elselabs.io
\n\033[0;33m=> Stopping development_nginx.1.3al3392m9fjk44nzgb5dwcifi  ------------------------------\033[0m
development_nginx.1.3al3392m9fjk44nzgb5dwcifi
\n\033[0;33m=> Removing development_nginx.1.3al3392m9fjk44nzgb5dwcifi  ------------------------------\033[0m
development_nginx.1.3al3392m9fjk44nzgb5dwcifi
\n REFRESH COMPLETE \n

I allowed the container to remain on the server so I could inspect the logs within it. They look like this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate for dev.elselabs.io
An unexpected error occurred:
No order for ID 241139202797
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
I'm running NGINX in a container. nginx:stable-alpine

The operating system my web server runs on is (include version):
I'm running this on Ubuntu 22.

My hosting provider, if applicable, is:
This infrastructure runs within a DigitalOcean droplet

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
I should be running the latest certbot dockerhub image.

2

Please don't use this option. It's not required for regular renewals.

The error comes from the validation server:

But I have no clue why Certbot, when run through cron, would request an invalid order? That's weird.

There should be a lot more debugging info present in the log. Could you please share the entire file?

4 Likes
3

There are a few cases where "No order for ID" on this forum have popped up due to requests being sent to a different Let's Encrypt server that didn't have a quite-up-to-date replication. Here's a recent example:

But usually retrying would fix it.

4 Likes
4

But that wouldn't explain why it wouldn't work with and would work without cron, right?

1 Like
5

Hi @Osiris and @petercooperjr,

I use the --force-renewal flag as the intention is to run the CRON prior to expiration, every couple months.

Since I run certbot via docker, the container is down after running the command. I cannot seem to access those logs you mentioned, unless you know of a way?

As I mentioned, I can run the script fine and it passes when run through the terminal - not the case with the CRON.

Thank you for your prompt responses.

1 Like
6

Usually one runs Certbot with the certbot renew command twice a day without the --force-renewal option: Certbot will decide if it's time to renew or not. Once the expiry date is within 30 days, Certbot starts renewing. There's not really any good reason to do this all yourself: just let Certbot do its job.

Sure: just add /var/log/letsencrypt to your Docker configuration as a volume, just like the rest? If not only temporary.

4 Likes
7

Thank you for your prompt response.

I've added the path as a volume, and it seems to be writing their fine.

As I mentioned, there does not seem to be a pattern as to when this command fails as a CRON other than noticing it fails after a few days from the initial setting. I'll have it renew over the weekend and follow up Monday.

Thank you for your help.

2 Likes
8

It has already been mentioned to run certbot more often. And, this isn't causing your primary problem. But, as you work on your final configuration please see below FAQ topic on why '0 0 ...' especially should be avoided.

3 Likes
9

Oh! That explains everything. Probably that's when Let's Encrypt is most under load, so that's when you're most likely to hit a case where the replication of your order isn't up-to-date.

Yes, use certbot correctly, where it controls whether to renew or not (never "forcing" renewal), and with a random time, and I'm pretty sure it'll fix all your problems.

4 Likes
10

I understand, so if I run it daily (not at 0 0), the only way for me to know if it's working fine, is if it continues to renew every few months?

My thought process was that if I can demand a renewal, then I can assume it should work whenever.

11

The whole point is to automate it so that it will work whenever, yes.

Feel free to put a reminder on your calendar to check on it 30 days before expiry. And for any sort of production system, you should have outside monitoring checking to make sure your site is accessible and that the certificate isn't close to expiration.

5 Likes
12

Got it.

So, running the script without force renewal brings up a prompt to be interacted with via terminal:

\n BEGINNING REFRESH \n
02/02/2024 20:59:34
\n\033[0;33m=> Requesting CERT for dev.elselabs.io ------------------------------\033[0
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Certificate not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/dev.elselabs.io.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Keep the existing certificate for now
2: Renew & replace the certificate (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): An unexpected error occurred:
EOFError
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
\n\033[0;33m=> Stopping development_nginx.1.kp23zhf48kagzvr61jy8vobb2  ------------------------------\033[0m
development_nginx.1.kp23zhf48kagzvr61jy8vobb2
\n\033[0;33m=> Removing development_nginx.1.kp23zhf48kagzvr61jy8vobb2  ------------------------------\033[0m
development_nginx.1.kp23zhf48kagzvr61jy8vobb2
\n REFRESH COMPLETE \n

I'm assuming if the cert does require renewal, it will do so without requiring user input?

13

While I agree with you both that using 0 0 should be avoided, Certbot should wait for a random amount of time before it would actually start renewing. But I'm not sure if running Certbot in Docker would mess up the whole non-interactive detection code :thinking:

You're using the run subcommand in your Docker code. For renewals, you should use the certbot renew subcommand. run should only be used for getting/installing new certificates.

You can test renewing with the --dry-run option. But that might not detect spurious problems with the production environment.

2 Likes
14

Your script should be running certbot renew, which should just handle everything already.

Yeah, it looks like it's getting confused and thinking it's running interactively. You might want to add --non-interactive

4 Likes
15

Sure, and that helps the CA for sure. But, do you know the range of this random sleep?

Is it, say, zero seconds to 480 seconds which could still randomly put you at spike periods

2 Likes
16

Between 1 second and 8 minutes:

Not a very long period if you'd ask me though.

2 Likes
17

A clarifying question.

I've switched to using renew instead. By doing so, at what point does certbot deem it necessary to renew the certificate? @petercooperjr mentioned to check 30 days before expiration, is that when I should expect the certbot renew to execute?

18

The Let's Encrypt recommendation is to renew with 1/3 life left. And, all LE certs today are 90 days so renewal with 30 days left. The duration of the cert may be different in future so best practice is to check for 1/3 life left which is what Certbot renew does (by default).

2 Likes
19

Certbot checks the expiry date of certificates every time you try to renew. It's by default configured to begin renewing 30 days before expiry.

Currently it's recommended to run certbot renew at least twice a day. This frequency might even increase once Automated Certificate Management Environment (ACME) Renewal Information (ARI) Extension becomes active.

4 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.