Unable to renew via cron

Help
1

I setup a certificate on a AWS bitnami server and running the renew script manually works successfully. The next logical step was to create a cron job as root in order to have this renewal happen automatically. I setup the script to run weekly and have had no shortage of problems - it has never worked running from cron.

The last investigation led me to ensure that the cron environment is the same as the machine's root environment. I ensured this is the case by loading the .profile at the start of my renewal bash script. After discovering it still fails I checked the output logs and each of the last 3 runs has produced a different error. I'm at my wits end here.

My domain is:

I ran this command:
/opt/bitnami/letsencrypt/lego --path="/opt/bitnami/letsencrypt" --email="" --domains="packtrackapp.com" --tls --http --http-timeout 30 --http.webroot /opt/bitnami/apps/letsencrypt renew

This command works fine when run manually as root. However in a cron job it produced these 3 separate failures:

2024/03/10 00:00:04 [INFO] [packtrackapp.com] acme: Trying renewal with 713 hours remaining
2024/03/10 00:00:04 [INFO] [packtrackapp.com, www.packtrackapp.com] acme: Obtaining bundled SAN certificate
2024/03/10 00:00:05 [INFO] [packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/324441185367
2024/03/10 00:00:05 [INFO] [www.packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/324441185377
2024/03/10 00:00:05 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/324441185367
2024/03/10 00:00:05 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/324441185377
2024/03/10 00:00:06 error: one or more domains had a problem:
acme: error: 0 :: POST :: https://acme-v02.api.letsencrypt.org/acme/authz-v3/324441185367 :: urn:ietf:params:acme:error:rateLimited :: Service busy; retry later., url:
Syntax OK

2024/03/17 00:00:05 [INFO] [packtrackapp.com] acme: Trying renewal with 545 hours remaining
2024/03/17 00:00:05 [INFO] [packtrackapp.com, www.packtrackapp.com] acme: Obtaining bundled SAN certificate
2024/03/17 00:00:06 [INFO] [packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/327147503317
2024/03/17 00:00:06 [INFO] [www.packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/327147503327
2024/03/17 00:00:06 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/327147503317
2024/03/17 00:00:06 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/327147503327
2024/03/17 00:00:06 error: one or more domains had a problem:
acme: error: 404 :: POST :: https://acme-v02.api.letsencrypt.org/acme/authz-v3/327147503317 :: urn:ietf:params:acme:error:malformed :: No such authorization, url:
Syntax OK

2024/03/24 00:00:04 [INFO] [packtrackapp.com] acme: Trying renewal with 377 hours remaining
2024/03/24 00:00:04 [INFO] [packtrackapp.com, www.packtrackapp.com] acme: Obtaining bundled SAN certificate
2024/03/24 00:00:04 [INFO] [packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/329910549437
2024/03/24 00:00:04 [INFO] [www.packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/329910549447
2024/03/24 00:00:04 [INFO] Unable to get the authorization for: https://acme-v02.api.letsencrypt.org/acme/authz-v3/329910549437
2024/03/24 00:00:05 [INFO] Unable to get the authorization for: https://acme-v02.api.letsencrypt.org/acme/authz-v3/329910549447
2024/03/24 00:00:05 error: one or more domains had a problem:
failed to post JWS message: failed to sign content: failed to sign content: square/go-jose: Error generating nonce: failed to get nonce from HTTP HEAD: 503 ::HEAD :: https://acme-v02.api.letsencrypt.org/acme/new-nonce :: unexpected end of JSON input ::
Syntax OK

The following is the output when I run my exact same renewal script manually, not with cron:

2024/03/28 16:58:08 [INFO] [packtrackapp.com] acme: Trying renewal with 264 hours remaining
2024/03/28 16:58:08 [INFO] [packtrackapp.com, www.packtrackapp.com] acme: Obtaining bundled SAN certificate
2024/03/28 16:58:09 [INFO] [packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/329910549437
2024/03/28 16:58:09 [INFO] [www.packtrackapp.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/329910549447
2024/03/28 16:58:09 [INFO] [packtrackapp.com] acme: use tls-alpn-01 solver
2024/03/28 16:58:09 [INFO] [www.packtrackapp.com] acme: use tls-alpn-01 solver
2024/03/28 16:58:09 [INFO] [packtrackapp.com] acme: Trying to solve TLS-ALPN-01
2024/03/28 16:58:10 http: TLS handshake error from 13.214.129.13:36520: EOF
2024/03/28 16:58:14 [INFO] [packtrackapp.com] The server validated our request
2024/03/28 16:58:14 [INFO] [www.packtrackapp.com] acme: Trying to solve TLS-ALPN-01
2024/03/28 16:58:16 http: TLS handshake error from 52.74.94.119:59222: EOF
2024/03/28 16:58:21 [INFO] [www.packtrackapp.com] The server validated our request
2024/03/28 16:58:21 [INFO] [packtrackapp.com, www.packtrackapp.com] acme: Validations succeeded; requesting certificates
2024/03/28 16:58:21 [INFO] [packtrackapp.com] Server responded with a certificate.

My web server is (include version):
Server version: Apache/2.4.34 (Unix)

The operating system my web server runs on is (include version):
4.4.0-1128-aws

My hosting provider, if applicable, is:
AWS

I can login to a root shell on my machine (yes or no, or I don't know):
YES

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot / certbot-auto n/a,
lego version 3.8.0 linux/amd64

2

Hello @jpodpro, welcome to the Let's Encrypt community. :slightly_smiling_face:

What are the environment difference between your cron job and your manually run job?

1 Like
3

Also note the Service busy in the message.

1 Like
4

regular root-user env:

SHELL=/bin/bash
TERM=xterm
OLDPWD=/home/bitnami
LC_ALL=en_US.UTF-8
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:.xspf=00;36:
SUDO_USER=bitnami
SUDO_UID=1000
USERNAME=root
MAIL=/var/mail/root
PATH=/opt/bitnami/apps/wordpress/bin:/opt/bitnami/varnish/bin:/opt/bitnami/sqlite/bin:/opt/bitnami/php/bin:/opt/bitnami/mysql/bin:/opt/bitnami/apache2/bin:/opt/bitnami/common/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PWD=/root
LANG=en_US.UTF-8
SHLVL=1
SUDO_COMMAND=/bin/su
HOME=/root
LANGUAGE=en_US.UTF-8
LOGNAME=root
LESSOPEN=| /usr/bin/lesspipe %s
SUDO_GID=1000
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/env

root cron job env:

SHELL=/bin/sh
LC_ALL=en_US.UTF-8
PATH=/opt/bitnami/apps/wordpress/bin:/opt/bitnami/varnish/bin:/opt/bitnami/sqlite/bin:/opt/bitnami/php/bin:/opt/bitnami/mysql/bin:/opt/bitnami/apache2/bin:/opt/bitnami/common/bin:/usr/bin:/bin
PWD=/root
LANG=en_US.UTF-8
SHLVL=1
HOME=/root
LANGUAGE=en_US.UTF-8
LOGNAME=root
_=/usr/bin/env

I had compared these last time and couldn't find anything that seemed like it would cause this. But I suppose it's something. I suppose I could manually set the exact same env vars?

5

Running at 00:00 is going to give you all sorts of problems, as you're not the only one to try to configure it at that time and they're all hammering Let's Encrypt's servers at once. Pick a different random time and you'll have much more success.

3 Likes
6

I see the PATH has fewer entries in the cron environment.

2 Likes
7

besides running . $HOME/.profile, how else can I get the env to be the same?

8

First see @petercooperjr comment.

1 Like
9

Sure, I will change that. But lets be real here. That is not the reason my env is different and only adequately explains the service busy failure, not the other 2.

10

The Service Busy is the first error, until that is fixed I don't really know that the other 2 errors are real.

1 Like
11

Well, no, but I'd bet that it's the cause of your problem.

They're really all rooted in Let's Encrypt servers being overloaded.

I think this is because the authorization objects being used weren't replicated between their servers fast enough between the request your client made (on one server) and the operation the client attempted to do on it (which hit another server in their cluster). (They call it "the 404 bug".)

This is really just another "Service Unavailable" error (503), just from a different part of the process.

2 Likes
12

Please take note of the dates. These 3 runs are each a week apart. They are not sequential immediate runs.

1 Like
13

Sorry @jpodpro , my bad! :frowning:

1 Like
14

Well thanks for all the suggestions. I will change the job run time and also add the missing PATH vars. Unfortunately I won't know for 2 months whether this worked. Unless anyone knows of a way to test this while being outside of the renew window?

2 Likes
15

You could adjust the DAYS option for the RENEW command (like DAYS=91 to force renew).

If you do that also adjust the SERVER option to use the Let's Encrypt staging server. The production server has stricter rate limits which can cause you to get locked out.

The staging server value is:

https://acme-staging-v02.api.letsencrypt.org/directory

Do be careful that you will get a cert but it is from the LE test system so will not validate if used with a production web service. So, be careful to take precautions to avoid it being used for such.

For lego options, see and especially the RENEW tab
https://go-acme.github.io/lego/usage/cli/options/

2 Likes