Having troubles with creating new certs and renewing on my server this morning. I ended up deleting the expired cert from my system and have started fresh to create a new one. Using test mode as I can't get it to verify. A records are set correctly, never had an issue in the past with this, so not sure what's changed that's causing the domain to not be verified. Hope someone can help me out!
I ran this command: certbot certonly --authenticator webroot --webroot-path /var/www/vhosts/remsfoundation.org/html -n --test-cert --debug-challenges --agree-tos -m 'domains@nine10.ca' -d www.remsfoundation.org -d remsfoundation.org
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for remsfoundation.org
http-01 challenge for www.remsfoundation.org
Using the webroot path /var/www/vhosts/remsfoundation.org/html for all unmatched domains.
Waiting for verification...
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges
Failed authorization procedure. remsfoundation.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: 178.128.231.186: Invalid response from http://remsfoundation.org/.well-known/acme-challenge/9DzDK3jM5e9QcUAJwixLXkIT_CPCgTyOLiISEufBEIQ: 403, www.remsfoundation.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: 178.128.231.186: Invalid response from http://www.remsfoundation.org/.well-known/acme-challenge/CUoa0BXdoBEcSOoXikwMXTJ_zIwddvpc4yXKpW4i7XE: 403
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: remsfoundation.org
Type: unauthorized
Detail: 178.128.231.186: Invalid response from
http://remsfoundation.org/.well-known/acme-challenge/9DzDK3jM5e9QcUAJwixLXkIT_CPCgTyOLiISEufBEIQ:
403
Domain: www.remsfoundation.org
Type: unauthorized
Detail: 178.128.231.186: Invalid response from
http://www.remsfoundation.org/.well-known/acme-challenge/CUoa0BXdoBEcSOoXikwMXTJ_zIwddvpc4yXKpW4i7XE:
403
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
Could not get cert. Aborting.
My web server is (include version): Apache/2.4.29
The operating system my web server runs on is (include version): Ubuntu 18.04.3 LTS (Bionic Beaver)
My hosting provider, if applicable, is: Digital Ocean
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0
First, a warning. You said you deleted the expired cert but if you now restart Apache it will fail to start since the files don't exist.
Your certbot version 0.31 is fairly old and the error message formats have changed. But, it looks like you get an http error code 403 Forbidden to the request by the Let's Encrypt server. Upgrade instructions are here for when you get time.
Those 403 would come from your server config denying access. The odd thing is I cannot reproduce it. I try a similar request and get the expected http 404 Not Found. Could you have changed anything since you posted?
Do you see the 403 or 404 in your Apache access logs?
curl -I http://www.remsfoundation.org/.well-known/acme-challenge/TEST0BXdoBEcSOoXikwMXTJ_zIwddvpc4yXKpW4i7XE
HTTP/1.1 404 Not Found
Date: Mon, 27 Jun 2022 17:41:06 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1
I see a different set of http response headers for your "home" page. Nothing inherently wrong with that but is that a clue to finding where the 403 is coming from? Notice even the content-type is different (UTF not ISO). Again, these headers would not cause a Let's Encrypt problem. Just something I see that I normally don't with such requests.
Hi Mike,
Thanks for the reply! I do see the Permission denied when viewing my error logs after attempting the challenge.
[Mon Jun 27 20:00:28.947984 2022] [core:error] [pid 25232:tid 139917745186560] (13)Permission denied: [client 3.15.158.249:21522] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/8kzhgnFyW8Crg3dHJvJpKAAFqkTEh0YVYEc2olvK9yo
[Mon Jun 27 20:00:28.952520 2022] [core:error] [pid 24909:tid 139917979789056] (13)Permission denied: [client 3.15.158.249:21524] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/wby98UL6EyvnG3myx9tVCR8oSYL321n32001kLsZ35o
[Mon Jun 27 20:00:29.003066 2022] [core:error] [pid 25064:tid 139917988181760] (13)Permission denied: [client 34.222.140.244:28908] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/8kzhgnFyW8Crg3dHJvJpKAAFqkTEh0YVYEc2olvK9yo
[Mon Jun 27 20:00:29.016176 2022] [core:error] [pid 25232:tid 139917862618880] (13)Permission denied: [client 18.157.73.0:57032] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/8kzhgnFyW8Crg3dHJvJpKAAFqkTEh0YVYEc2olvK9yo
[Mon Jun 27 20:00:29.055681 2022] [core:error] [pid 24909:tid 139917602576128] (13)Permission denied: [client 34.222.140.244:28910] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/wby98UL6EyvnG3myx9tVCR8oSYL321n32001kLsZ35o
[Mon Jun 27 20:00:29.079512 2022] [core:error] [pid 25232:tid 139917854226176] (13)Permission denied: [client 18.157.73.0:57034] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/wby98UL6EyvnG3myx9tVCR8oSYL321n32001kLsZ35o
[Mon Jun 27 20:00:29.147999 2022] [core:error] [pid 25064:tid 139917753513728] (13)Permission denied: [client 66.133.109.36:21616] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/wby98UL6EyvnG3myx9tVCR8oSYL321n32001kLsZ35o
[Mon Jun 27 20:00:29.213299 2022] [core:error] [pid 24909:tid 139917711615744] (13)Permission denied: [client 66.133.109.36:55210] AH00132: file permissions deny server access: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/8kzhgnFyW8Crg3dHJvJpKAAFqkTEh0YVYEc2olvK9yo
I am aware of the restart apache issue and have ensured that my ssl.conf file for the domain is no longer active so I have been able to restart apache2. I did inherit this server a few months ago, so apologies if I'm not 100% sure on some of the setups here.
Definitely a permissions issues. If I create this file as root: /var/www/vhosts/remsfoundation.org/html/.well-known/acme-challenge/wby98UL6EyvnG3myx9tVCR8oSYL321n32001kLsZ35o I get the 403 forbidden Error when I try to access, when I change the file ownership to remsfoundation it works. Is there something I can do to ensure the challenge files are created with the correct permissions?
Hmm. When I use sudo certonly webroot I get permissions of 'root root' but -rw-r--r-- so everyone should have read access.
I have never seen the need for instructing certbot to use different permissions. You could use a script to run certbot that does the chown after. But, has anything changed b/c this has been running a long time it looks.
I'd use another folder [far from /var/www/vhosts/*].
A dedicated folder - just for challenge requests.
Then it can have whatever access is needed [or more].
I've got about 150 sites setup this way, so I'd prefer not to have to change the setup for renewing certs on this server as it's been running well for years. Nothing has changed on this server to suddenly start this issue as far as I'm aware. This is coming out of the blue today.
Clearly something has changed. Your version of certbot is ancient so doesn't look like it changed either.
When you run certbot, it pauses when you use debug-challenges. Can you look at the challenge file permissions to see what it says?
Normally certbot is run as root. I notice you did not include sudo in the first post. Should you be? Or, could your org use an unusual setup such that you run it under authority of remsfoundation user? The private key files are usually locked down to just root/root and -rw-------- for security but maybe your org does it differently?
Could you have a new security system active like AppArmor or something?
I'm not really sure what to suggest. And, I can't think of what acme component could have changed to cause this.
-rw-rw----+ 1 root root 87 Jun 28 02:10 GaBE53ummRx1h094B806NuyfSkC1GIuh7gTtuVg9-7I
-rw-rw----+ 1 root root 87 Jun 28 02:10 iUCFuMvLE47MAhbD87B9lQdrLRlM48Z8SMbQ0j25U1w
I am running certbot as root, I actually had to move the remsfoundation site to a different server as it was down for too long, but I have other sites with the same issue. In this case its the domain inthezonepartyrentals.com - so it doesn't seem to be user specific.
Huh. The + sign means the file has ACL permissions set. certbot would not do that. I have never used those myself. Maybe start with this ubuntu thread about them?
I'm guessing you have some sort of security package managing these.
At least we now know why the 403 happens. The lack of r for everyone which is what certbot would have set by default. You either need to let certbot set default perms or figure out how to set better ACL for the challenge files.
