Certbot not renewing, challenge fails with empty string

I have taken over a Digital Ocean server at work by an employee who is no longer here and some of our certificates are not renewing. I’m not really a ‘server guy’ so this is out of my comfort zone and I could use some pointers.

I know all of the certificates were being renewed in the past as I’d integrated several StatusCake tests with our team’s Discord server. I would see the upcoming expiration notice and then a successful renewal. This has stopped recently and two domains expired and I can’t renew them successfully. I’ll fill out the questions here and provide as much detail as I can.

I’m concerned that the challenges which fail are failing with an empty string. Several domains are showing this. I’ve provided the output below.

My domain is: www.keylogic.ca keylogic.ca

I ran this command: certbot renew

It produced this output:

Attempting to renew cert (keylogic.ca) from /etc/letsencrypt/renewal/keylogic.ca.conf produced an unexpected error: Failed authorization procedure. www.keylogic.ca (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge “AP5zJP33NVm6GlP-RBXISaeZTys-9O4wOfHl1czGf9Q.E1QS09vT95l-xxWbiWnUP24ePZ9EgKsRo_YTObqj4eE” != “”, keylogic.ca (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: The key authorization file from the server did not match this challenge “YZ1-JXd4WOkaVCi9XIYIfpnzkZpPbIKVMOG86CLi6K8.E1QS09vT95l-xxWbiWnUP24ePZ9EgKsRo_YTObqj4eE” != “”. Skipping.

Domain: keylogic.ca
Type: unauthorized
Detail: The key authorization file from the server did not match
this challenge
“YZ1-JXd4WOkaVCi9XIYIfpnzkZpPbIKVMOG86CLi6K8.E1QS09vT95l-xxWbiWnUP24ePZ9EgKsRo_YTObqj4eE”
!= “”

My web server is (include version): Apache 2.4.29

OS: Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-99-generic x86_64)

My hosting provider, if applicable, is: Amazon

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.31.0

Hi @patrickatkeylogic

your webserver configuration is buggy.

Checking

http://www.keylogic.ca/.well-known/acme-challenge/1234

there isn't an error / http status 404 Not Found, instead there is a http status 200 - and an empty result.

Why?

Fix that.

Thanks for the reply.

I think we should see the token when going to that URL? I’m betting one of the redirects is buggy (we redirect HTTP traffic to HTTPS). I also don’t see a .well-known directory anywhere in the web root so that might be another issue.

That's the expected result, yes, when certbot is being ran. The 1234 test file should result in a 404 file not found error though.

That's not a webserver. A valid answer could have been "Apache", "nginx", "openresty", "Varnish" among others.

Seems to be an issue in your openresty configuration.

Whoops, I’ll update the post. It’s Apache/2.4.29.

I’ll dig into the redirects. Thanks for the help.

I'm only seeing openresty and/or Varnish in the HTTP headers?

That actually might be the crux of the issue. The failing URLs are now pointing to a webflow site that a contractor built recently. I should have seen it earlier but it happened pretty recently.

Thanks again for the help.