After a successful automated certificate renewal, apache server is not working during a moment

Help
1

Hello,

I have 2 ubuntu 24.04 LTS servers with Let's Encrypt on them - each with a web app, based on apache server, one with MariaDB the other with PostgreSQL databases. Everything is fine ... except the day the certificate is renewd, I saw a strange and not understandable behaviour - for me - let me give a bit of context.

Each day, the following tasks are executed :
00:01 CET : backup of the database
00:11 CET : stop of the server (apache server, and database)
00:12 CET : renew certificate, if needed
00:31 CET : update and upgrade of the linux server
00:59 CET : start of the server (database, and apache server)

All thes tasks are executing normally everyday, everything is under control for more than 2.5 years for one server, 1.5 year for the other server; the detailed logs proved that everything is right ... except the day when the certificate is renewd: that day, even if the 00:59 CET task log is fine, the web server is not actually running. More strange, a few hours later, when manually I stop and restart the server, then it is working fine. It seems that there is a ghost in the machine...

Any clue / ideas about that?
Has someone in this forum already faced a similar issue / strange behaviour?

Thanks for any information,
BR,

Guy.

My domain is: is it mandatory to fill that field?

My web server is (include version): 2.4.58-1

The operating system my web server runs on is (include version): ubuntu server 24.04 LTS

My hosting provider, if applicable, is: OVH

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.9.0

2

Has that happened the entire time or has it just started?

What was the original command you used to get the cert? If you don't remember please show contents of the renewal config file in /etc/letsencrypt/renewal. Need to know if the --apache plugin is being used and any --deploy-hooks.

Note that a Certbot renew usually doesn't do much of anything. Each renew run inspects the local cert but if it has enough life left Certbot is finished. (your version of Certbot would only look at the days remaining given Let's Encrypt no longer supports OCSP but that's a sideshow to this problem).

I don't know the details of what the CET task log shows but clearly it does not represent that active state of Apache. Using something like below in that sequence might help. But, given this only happens every 60 days or so it is not particularly helpful now :slight_smile:

 sudo systemctl status -l --no-pager apache2
2 Likes
3

Mike, thanks for you reply and your points for investigation.

The answers:

  • Yes, that behaviour happens every time the certificate is renewed (once every 60 days)

  • Honestly, I don't remember exactly. In the /etc/letsencrypt/renewal/.conf, there are the following entries:
    authenticator = apache
    installer = apache

  • I will try to capture the state of apache with the command "systemctl status ..." you gave, fyi the next renewal for one server is tomorrow or the day after, so it will be near; for the other server, it will be in 3 or 4 days, so also another analysis would be done.

Thanks & BR,

Guy.

4

Ah, so even though you stopped Apache Certbot will start it when it needs to renew. Not every run, just when an actual cert renew is needed. It does not stop it after so it would remain running.

Certbot uses a default command based on the o/s. Do you have any special commands to start/stop it? Perhaps the Certbot method isn't right for your system.

Or, perhaps if Certbot starts Apache and your final step starts the DB and Apache there is some kind of sequencing issue? There are several options if this is the case. Switching to standalone method is one, running Certbot outside of this sequence so it can just rely on the running Apache is another (using certonly --webroot option instead of --apache), or just stopping Apache always after Certbot to ensure your last startup step always has the same initial state.

2 Likes
5

Mike,

Thanks for the elements: I will review the daily jobs to be sure that apache server situation is relevant at any moment, whatever the circumstances.

BR,

Guy.

1 Like
6

So, I changed the sequence of daily tasks :Each day, the following tasks are executed :
00:01 CET : backup of the database
00:06 CET : stop of the server (apache server, and database)
00:11 CET : update and upgrade of the linux server
00:36 CET : start of the server (database, and apache server)
00:37 CET : renew certificate, if needed

That way, the certificate is renewd with Apache server fully working
Today was the day of reneal of the certificate, and everything performs well as expected.

Thanks Mike for the exchanges and for teaching me that certbot renewal needs Apache server to be running.

Hope these exchanges will be helpful...

BR,

Guy.

2 Likes