After a successful automated certificate renewal, apache server is not working during a moment

Help
1

Hello,

I have 2 ubuntu 24.04 LTS servers with Let's Encrypt on them - each with a web app, based on apache server, one with MariaDB the other with PostgreSQL databases. Everything is fine ... except the day the certificate is renewd, I saw a strange and not understandable behaviour - for me - let me give a bit of context.

Each day, the following tasks are executed :
00:01 CET : backup of the database
00:11 CET : stop of the server (apache server, and database)
00:12 CET : renew certificate, if needed
00:31 CET : update and upgrade of the linux server
00:59 CET : start of the server (database, and apache server)

All thes tasks are executing normally everyday, everything is under control for more than 2.5 years for one server, 1.5 year for the other server; the detailed logs proved that everything is right ... except the day when the certificate is renewd: that day, even if the 00:59 CET task log is fine, the web server is not actually running. More strange, a few hours later, when manually I stop and restart the server, then it is working fine. It seems that there is a ghost in the machine...

Any clue / ideas about that?
Has someone in this forum already faced a similar issue / strange behaviour?

Thanks for any information,
BR,

Guy.

My domain is: is it mandatory to fill that field?

My web server is (include version): 2.4.58-1

The operating system my web server runs on is (include version): ubuntu server 24.04 LTS

My hosting provider, if applicable, is: OVH

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.9.0

2

Has that happened the entire time or has it just started?

What was the original command you used to get the cert? If you don't remember please show contents of the renewal config file in /etc/letsencrypt/renewal. Need to know if the --apache plugin is being used and any --deploy-hooks.

Note that a Certbot renew usually doesn't do much of anything. Each renew run inspects the local cert but if it has enough life left Certbot is finished. (your version of Certbot would only look at the days remaining given Let's Encrypt no longer supports OCSP but that's a sideshow to this problem).

I don't know the details of what the CET task log shows but clearly it does not represent that active state of Apache. Using something like below in that sequence might help. But, given this only happens every 60 days or so it is not particularly helpful now :slight_smile:

 sudo systemctl status -l --no-pager apache2
2 Likes
3

Mike, thanks for you reply and your points for investigation.

The answers:

  • Yes, that behaviour happens every time the certificate is renewed (once every 60 days)

  • Honestly, I don't remember exactly. In the /etc/letsencrypt/renewal/.conf, there are the following entries:
    authenticator = apache
    installer = apache

  • I will try to capture the state of apache with the command "systemctl status ..." you gave, fyi the next renewal for one server is tomorrow or the day after, so it will be near; for the other server, it will be in 3 or 4 days, so also another analysis would be done.

Thanks & BR,

Guy.

4

Ah, so even though you stopped Apache Certbot will start it when it needs to renew. Not every run, just when an actual cert renew is needed. It does not stop it after so it would remain running.

Certbot uses a default command based on the o/s. Do you have any special commands to start/stop it? Perhaps the Certbot method isn't right for your system.

Or, perhaps if Certbot starts Apache and your final step starts the DB and Apache there is some kind of sequencing issue? There are several options if this is the case. Switching to standalone method is one, running Certbot outside of this sequence so it can just rely on the running Apache is another (using certonly --webroot option instead of --apache), or just stopping Apache always after Certbot to ensure your last startup step always has the same initial state.

2 Likes
5

Mike,

Thanks for the elements: I will review the daily jobs to be sure that apache server situation is relevant at any moment, whatever the circumstances.

BR,

Guy.

1 Like
6

So, I changed the sequence of daily tasks :Each day, the following tasks are executed :
00:01 CET : backup of the database
00:06 CET : stop of the server (apache server, and database)
00:11 CET : update and upgrade of the linux server
00:36 CET : start of the server (database, and apache server)
00:37 CET : renew certificate, if needed

That way, the certificate is renewd with Apache server fully working
Today was the day of reneal of the certificate, and everything performs well as expected.

Thanks Mike for the exchanges and for teaching me that certbot renewal needs Apache server to be running.

Hope these exchanges will be helpful...

BR,

Guy.

2 Likes
7

As others have said, you need a running webserver to update your certificates. But not only don't need to shut your webserver down, you don't even need to restart it after updating your certificates.

Apache has a "graceful" option that allows you reload your updated certificate without restarting. Check this description out. I'm on FreeBSD, and I use service apache24 reload to gracefully reload the new certificate.

I've never used Ubuntu, but apparently systemctl reload apache2 should remove the need to restart the webserver all together.

EDIT: Here's how it looks when I run the command.

root@ravage:~# service apache24 reload
Performing sanity check on apache24 configuration:
Syntax OK
Performing sanity check on apache24 configuration:
Syntax OK
Performing a graceful restart
root@ravage:~#

And Apache never stops working.

8

Yes, it is a fair point that only an Apache graceful reload is required after getting a fresh cert.

But, in this case they used the --apache plugin option. That option automatically does a graceful reload so no further action is needed.

For people using Certbot (as in this thread) but using option of certonly --webroot they can, and should, add a graceful reload with a --deploy-hook.

The original problem here related to some sort of conflict during a longer system update sequence with Apache down. Certbot --apache also starts Apache and leaves it running which apparently caused some conflict with their longer system update workflow.

4 Likes