Certbot renewals have suddenly started failing

Well then unless you mitigate whatever did that, it will do it again (no doubt).
The Internet never sleeps!
[It is ALWAYS on the attack]

I would also check on maximum-keep-alives

That's something @fmouse needs to figure out indeed, but not solely for the sake of certbot.

Once that issue has been addressed (somehow), certbot will be fine.

But certbot would also be fine with --webroot
And that would always work, despite any consequence or side-effect of the suspected DoS attacks.

I can't agree with you more.
But I am here to do my best to make certbot work.
And that impediment can easily be ignored (not fixed by us - that is not my goal here).

My guess at the apachectl failure is likely due to an unhandled timeout exceeded situation.
I suppose that things are expected to gracefully close within a certain amount of time.
But they don't and it tries to proceed without actually having completed that task.
Again (because I have said this in this forum many many times), Apache is notorious for running at all costs.
And this is likely just one more example of that happening.

Yes, the issue may recur, of course. My company has been an IPP since the 90s, and I've seen a lot of crap. Yes, I'll need to be more vigilant. The quick-fix is to do a more thorough killing of apache when fail2ban detects a DoS attack. I worked for one of Austin's first ISPs in the mid-90s, when no one knew what "the Internet" even was, and I cut my teeth with some great UNIX gurus there. UNIX is a tinker-toy and you can put pieces together to build just about anything you want to.

I know that when I use certbot to renew a wildcard cert, it does need a full shutdown of the running server. I well recall reading that. I'll research your statement that it doesn't do this when doing a normal certbot renewal. For some reason the stuck children were poisoning the soup and clearing that made everything good again.

Again, what I proposed (twice now) isn't trying to fix any of those apache deficiencies.
It is merely my way of trying to build you a system that works without exception.

You need cert.
You get a cert.
[but apache is broken and some processes haven't restarted in months]
You still get a cert!

Exactly. The timeout was with certbot, I would guess, which choked on the stuck processes.

And that would be a very logical thing for certbot to do, for obvious security reasons.

No. I suspect the timeout was within apachectl.
Which returned no error to certbot, even though it never actually shutdown/restarted ALL of the processes.

It's then a hit or miss for LE trying to reach the new processes to get the correct response.
[any old processes would fail]

Could well make sense.

Maybe something like this might reduce your problems (automatically):

KeepAlive On
MaxKeepAliveRequests 3
KeepAliveTimeout 9
MaxRequestsPerChild  100000

I keep forgetting this is NOT an apache forum!

OR
You could switch to NGINX (FTW!)

There I go again, helping people with things that have nothing to do with this forum!
My bad...

I would really try using the --webroot method; as that won't require restarting apache.

Heh!! I'm 80 years old - I don't have time to learn nginx. Thanks for the KeepAlive suggestions. That might well help. I'll research it. I did a full reboot of my server a few days ago, and a couple of things went slightly south - which is unusual since the folks at Linode are pretty sharp. It ain't the kinder, gentler Internet I signed up for in 1995.

My hat's off to you!

Can you get the cart back upright?

I see that one of your sites' cert expired today and hasn't yet been renewed:
https://www.fmp.com/

     	 rg305 Community leader         

August 31

fmouse:
I'm 80 years old

My hat's off to you!

fmouse:
I did a full reboot of my server a few days ago, and a couple of
things went slightly south

Can you get the cart back upright?

Hey, I'm not THAT old! I've been getting the cart back upright on the
Internet for about 25 years. I've been programming in multiple
languages since a lot of young sysadmins and programmers were wearing
diapers.

I'm also a musician, mostly retired, but still famous enough in a few
circles to be involved - and I'm leaning more on that these days. I
tend to let my very small IPP business run on auto-pilot.

I see that one of your sites' cert expired today and hasn't yet been
renewed:
https://www.fmp.com/

Hmmm. Yes, I added a security exception for that, but I see I need to
do a bit more work. I'm going to have to manually redo some of the
certs that have been bollixed.

Thanks for the heads-up. I had fmp.com configured to use a wildcard
cert, good until the end of October, but stepped on the apache config
trying to solve the problem today. Fixed.

I like it that way too :slight_smile:

Pretty much what's already globally set on the server

Not supported on Apache 2.4

You're right:
MaxRequestsPerChild has been renamed to MaxConnectionsPerChild

That's just a whole bunch of b*llsh*t! There are a lot of terrible guides out there. Don't automatically believe everything you read. Do your own research. Try to understand every step to the fullest. I wouldn't expect less from someone who worked with the first baby steps of the internet...

A wildcard certificate requires the DNS-01 challenge, which does not even use a webserver! Nor should it ever be necessary to shutdown an entire server...