Unable to renew certs for non http services, web sites only https allowed

#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
klam.ca
I ran this command:
certbot renew --webroot --webroot-path /var/www/default
It produced this output:
+++++++++++++
this is one example,
domains:
dav.klam.ca – no web page, webdav server
davical.klam.ca – no web page, caldav/carddav server
mail.klam.ca – has attached entries fro smtp, submission, imaps
???.klam.ca – postgresql & postfixadmin - admin functions only accessible from lan
www.klam.ca – only https
As HTTP is currently in the process of being deprecated in favour of HTTPS , port 80 if blocked.
+++++++++++++
below is the logout for one of the sub domains

Processing /etc/letsencrypt/renewal/dav.klam.ca.conf

Cert is due for renewal, auto-renewing…
Non-interactive renewal: random delay of 238 seconds
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dav.klam.ca
Using the webroot path /var/www/default for all unmatched domains.
Cleaning up challenges
Encountered exception during recovery:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 75, in handle_authorizations
resp = self._solve_challenges(aauthzrs)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 139, in _solve_challenges
resp = self.auth.perform(all_achalls)
File “/usr/lib/python3/dist-packages/certbot/plugins/webroot.py”, line 85, in perform
return [self._perform_single(achall) for achall in achalls]
File “/usr/lib/python3/dist-packages/certbot/plugins/webroot.py”, line 85, in
return [self._perform_single(achall) for achall in achalls]
File “/usr/lib/python3/dist-packages/certbot/plugins/webroot.py”, line 210, in _perform_single
with open(validation_path, “wb”) as validation_file:
NotADirectoryError: [Errno 20] Not a directory: ‘/var/www/default/.well-known/acme-challenge/39s5kG7oF6uHJVlHX3BMyiYkm_21AjmOBgeYojMbBOI’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/error_handler.py”, line 108, in _call_registered
self.funcs-1
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 323, in _cleanup_challenges
self.auth.cleanup(achalls)
File “/usr/lib/python3/dist-packages/certbot/plugins/webroot.py”, line 224, in cleanup
os.remove(validation_path)
NotADirectoryError: [Errno 20] Not a directory: ‘/var/www/default/.well-known/acme-challenge/39s5kG7oF6uHJVlHX3BMyiYkm_21AjmOBgeYojMbBOI’
Attempting to renew cert (dav.klam.ca) from /etc/letsencrypt/renewal/dav.klam.ca.conf produced an unexpected error: [Errno 20] Not a directory: ‘/var/www/default/.well-known/acme-challenge/39s5kG7oF6uHJVlHX3BMyiYkm_21AjmOBgeYojMbBOI’. Skipping.

My web server is (include version):
apache2
The operating system my web server runs on is (include version):
Debian 10
My hosting provider, if applicable, is:
self
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.31.0

#2

Hi @zzz2002

sounds like you have created a file

/var/www/default/.well-known/acme-challenge

or

/var/www/default/.well-known

so it’s impossible to use the same name as directory name. Remove that file.

#3 
/var/www/default/.well-known/acme-challenge is a directory 
username & groupname is  www-data (standard Debian usage)
permissions were 755, I have tried changing the to 777, 770 ...
doesn't make any difference.
FYI
/var/www,   /var/www/default,   /var/www/default/.well-known,
/var/www/default/.well-known/acme-challenge  -
are all directories/subdirecories - - www-data/www-data 775

the file "39s5k ....... bBoi' does NOT exist on my system (anywhere).
#4

Did you run Certbot as root or with sudo?

#5

I run it as root, its a cron job, run about every 8 hours.

Sleep $((RANDOM \% 3600 + 1)) && certbot renew --webroot --webroot-path /var/www/default

I added acme-challenge as a sub-directory, giving me /var/www/default/.well-known/acme-challenge, but It made no difference.
So I have reverted to “/var/www/default/.well-known”

#6

What happens if you run a command like this?

touch /var/www/default/.well-known/acme-challenge/test.txt

#7

For this test I added the directory acme-challenge, in my original setup which worked, I did not have it in my setup.
I get a new empty file in /var/www/default/.well-known/acm-challenge
The file details are – test.txt 644 root:root
If I edit the file with pluma or mcedit I can add/delete/modify the contents.

#8

I now see a Gandi parking page when trying to access that file.