Cron Renewal of Certs Not Working

Help
1

I have been unable to obtain cert renewal automatically. Manual renewal works great.

There appears to be a conflict because the system's init is systemd.

The latest attempt to fix the daily cron job to renew automatically is show below. The file is a mess.

I would appreciate any assistance. Thanks!

System Description:

Ubuntu 22.04 LTS (Web server, Reverse Proxy and IPTables firewall)
Apache2 2.4.52
Init System: systemd

I have full direct system access to system

root@genesis:/etc/cron.d# cat certbot
#/etc/cron.d/certbot: crontab entries for the certbot package

#Upstream recommends attempting renewal twice a day

#Eventually, this will be an opportunity to validate certificates
#haven't been revoked, etc. Renewal will only occur if expiration
#is within 30 days.

#Important Note! This cronjob will NOT be executed if you are
#running systemd as your init system. If you are running systemd,
#the cronjob.timer function takes precedence over this cronjob. For
#more details, see the systemd.timer manpage, or use systemctl show
#certbot.timer.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a ! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew
root@genesis:/etc/cron.d# ls -l

2

A couple things. If you got your cert with a --manual method it cannot be automated. There is no way to automate manual actions.

And, most Linux systems are systemd so that by itself is not a problem. Usually those systems have a timer instead of a cron job. See (this link) for how to check for that and more details.

We can figure this out. The best place to start is to show us the contents of the conf file for this cert in this folder

/etc/letsencrypt/renewal
6 Likes
3

If there are multiple cron or systemd timers jobs installed it's generally OK, because one will succeed and the others will do nothing.

What would be helpful is the error output, from a command like:

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin certbot renew --dry-run
6 Likes
4

I had a problem with running ./acme.sh by hand succeeding, but failing if invoked via cron (or a systemd timer). Turned out that the environment differed, and it was solved by adding environment into the cron invocation along the lines of

#!/bin/bash
(
NSUPDATE_SERVER=192.168.1.2 NSUPDATE_KEY=pwd/okiefrog.key ./acme.sh
...
)

in a script invoked via cron. As a old stubborn fellow, I find systemd timers to be abominable and regularly stomp them out of existence ...

7 Likes
5

Hi MikeMcQ

per your request:

root@genesis:/etc/letsencrypt/renewal# cat allenintech.com-0002.conf
#renew_before_expiry = 30 days
version = 1.31.0
archive_dir = /etc/letsencrypt/archive/allenintech.com-0002
cert = /etc/letsencrypt/live/allenintech.com-0002/cert.pem
privkey = /etc/letsencrypt/live/allenintech.com-0002/privkey.pem
chain = /etc/letsencrypt/live/allenintech.com-0002/chain.pem
fullchain = /etc/letsencrypt/live/allenintech.com-0002/fullchain.pem

#Options used in the renewal process
[renewalparams]
account = 167f86e1432ceed045ef851b5bae590f
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa

1 Like
6

Thanks. And why do you think renewal is not working?

Because the cert for that domain was issued on Nov2 and would not renew until early January

Also, when we see '-0002' in the cert name it often points to problems with Apache config.

What does the command certbot certificates show?

7 Likes
7

Thanks for the info...
Could you share the output of:

openssl x509 -noout -dates -in /etc/letsencrypt/live/allenintech.com-0002/cert.pem | grep -e "notBefore" -e "notAfter="

My inquiry is based on @MikeMcQ comment:

5 Likes
8

(It is possible that the symlinks are hosed.)

6 Likes
9

To get the cert I had to issue CLI commands: certbot renew or certbot --apache. For the past several years I have only issued those commands. Years ago I issued a command that has now been deprecated. -0002 is not recent. It has been that way for years.

The next time the cert is due in January, I would like to have the cron file working. If it does not work, then I'll issue cerbot renew or certbot --apache.

10

@Rip

The output

/etc/letsencrypt/renewal# openssl x509 -noout -dates -in /etc/letsencrypt/live/allenintech.com-0002/cert.pem | grep -e "notBefore" -e "notAfter="
notBefore=Nov 2 02:06:18 2022 GMT
notAfter=Jan 31 02:06:17 2023 GMT
root@genesis:/etc/letsencrypt/renewal#

1 Like
11

So the output doesn't show a renewal due yet. It is coming though, and you are wanting to "prebug" your cron or system timers... Yes?

6 Likes
12

Here are the results. All other sites were taken out of the system, except the following:
allenintech.com (reachable with cert)
nextcloud.allenintech.com (reachable with cert)
kibana.alleninech.com (reachable with cert)
elasticsearch.allenintech.com (purposely not reachable)

I used certbot renew and certbot --apache to obtain the certs. I do not stop apache2.

RESULTS OF YOUR REQUEST. Thanks!

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/actngop.org-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for actngop.org and www.actngop.org
Failed to renew certificate actngop.org-0001 with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/actngop.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/actngop.org.conf is broken.
The error was: expected /etc/letsencrypt/live/actngop.org/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/allenintech.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for www.allenintech.com and allenintech.com
Failed to renew certificate allenintech.com-0001 with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/allenintech.com-0002.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for allenintech.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/allenintech.com.conf is broken.
The error was: expected /etc/letsencrypt/live/allenintech.com/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gitlabrunner.allenintech.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for gitlabrunner.allenintech.com and www.gitlabrunner.allenintech.com
Failed to renew certificate gitlabrunner.allenintech.com-0001 with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gitlabrunner.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/gitlabrunner.allenintech.com.conf is broken.
The error was: expected /etc/letsencrypt/live/gitlabrunner.allenintech.com/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gitlabserver.allenintech.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for gitlabserver.allenintech.com and www.gitlabserver.allenintech.com
Failed to renew certificate gitlabserver.allenintech.com-0001 with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gitlabserver.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/gitlabserver.allenintech.com.conf is broken.
The error was: expected /etc/letsencrypt/live/gitlabserver.allenintech.com/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/guacamole.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for guacamole.allenintech.com and www.guacamole.allenintech.com

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: guacamole.allenintech.com
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up A for guacamole.allenintech.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for guacamole.allenintech.com - check that a DNS record exists for this domain

  Domain: www.guacamole.allenintech.com
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up A for www.guacamole.allenintech.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for www.guacamole.allenintech.com - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate guacamole.allenintech.com with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/hishandstn.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/hishandstn.org.conf is broken.
The error was: expected /etc/letsencrypt/live/hishandstn.org/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/kibana.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for kibana.allenintech.com
Failed to renew certificate kibana.allenintech.com with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/nextcloud.allenintech.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for nextcloud.allenintech.com and www.nextcloud.allenintech.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/nextcloud.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/nextcloud.allenintech.com.conf is broken.
The error was: expected /etc/letsencrypt/live/nextcloud.allenintech.com/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/tomcat.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for tomcat.allenintech.com and www.tomcat.allenintech.com

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: tomcat.allenintech.com
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up A for tomcat.allenintech.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for tomcat.allenintech.com - check that a DNS record exists for this domain

  Domain: www.tomcat.allenintech.com
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up A for www.tomcat.allenintech.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for www.tomcat.allenintech.com - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate tomcat.allenintech.com with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.actngop.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/www.actngop.org.conf is broken.
The error was: expected /etc/letsencrypt/live/www.actngop.org/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.elasticsearch.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for www.elasticsearch.allenintech.com and elasticsearch.allenintech.com
Failed to renew certificate www.elasticsearch.allenintech.com with error: Could not bind TCP port 80 because it is already in use by another process on this system (such as a web server). Please stop the program in question and then try again.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.kibana.allenintech.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for kibana.allenintech.com and www.kibana.allenintech.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following simulated renewals succeeded:
  /etc/letsencrypt/live/allenintech.com-0002/fullchain.pem (success)
  /etc/letsencrypt/live/nextcloud.allenintech.com-0001/fullchain.pem (success)
  /etc/letsencrypt/live/www.kibana.allenintech.com/fullchain.pem (success)

The following simulated renewals failed:
  /etc/letsencrypt/live/actngop.org-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/allenintech.com-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/gitlabrunner.allenintech.com-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/gitlabserver.allenintech.com-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/guacamole.allenintech.com/fullchain.pem (failure)
  /etc/letsencrypt/live/kibana.allenintech.com/fullchain.pem (failure)
  /etc/letsencrypt/live/tomcat.allenintech.com/fullchain.pem (failure)
  /etc/letsencrypt/live/www.elasticsearch.allenintech.com/fullchain.pem (failure)

Additionally, the following renewal configurations were invalid: 
  /etc/letsencrypt/renewal/actngop.org.conf (parsefail)
  /etc/letsencrypt/renewal/allenintech.com.conf (parsefail)
  /etc/letsencrypt/renewal/gitlabrunner.allenintech.com.conf (parsefail)
  /etc/letsencrypt/renewal/gitlabserver.allenintech.com.conf (parsefail)
  /etc/letsencrypt/renewal/hishandstn.org.conf (parsefail)
  /etc/letsencrypt/renewal/nextcloud.allenintech.com.conf (parsefail)
  /etc/letsencrypt/renewal/www.actngop.org.conf (parsefail)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
8 renew failure(s), 7 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
root@genesis:/etc/letsencrypt/renewal#
1 Like
13

Yes, That is correct. I want to "prebug" the cron or system timers so the next time the cert is due the cron will work.

Thanks!

2 Likes
14

It is possible the symlinks are hosed. How would I fix them?

1 Like
15

The cron file says

#Important Note! This cronjob will NOT be executed if you are
#running systemd as your init system. If you are running systemd,
#the cronjob.timer function takes precedence over this cronjob. For
#more details, see the systemd.timer manpage, or use systemctl show
#certbot.timer.

I am running systemd

16

OK so let me have a minute. I have seen this before and I can't remember who gave the fix but let me look in my "bag".... Let's look at the symlink(s) first.

7 Likes
17

I appreciate the insight. I'll keep that in mind. Thanks!

1 Like
18

@Buffalo is excellent. Much knowledge there.

8 Likes
19

OK. I have certs for sites
allenintech.com

elasticsearch.allenintech.com
kibana.allenintech.com

I do not need the certs for the other sites such as guacamole.allenintech.com, etc.

I am trying to trouble shoot cron so that in January the certs are renewed automatically by cron.

1 Like
20

Ok fine. But would you show the output of

ls -la -R /etc/letsencrypt/

It will show the regular files and symlinks (which may need to be repaired.)

You may not need them now, but you will in the near future.

7 Likes