Create certificate : Error 403 since January

[christianlx]

Hi,
I’m searching for days a solution for my web server.

I use an apache Web Server on Centos 7.x and I encouter 403 issue since the LE update

This server already host 6 https websites configured with let’s encrypt in 2017. Since January, I cannot create anymore new certificate for new web sites whereas renewals on old ones are working.

I’ve always have this message :

Domain: www.hxxxxxxxxxxxxxxn.com
Type: unauthorized
Detail: Invalid response from
http://www.hxxxxxxxxxxxxxxxxxxn.com/.well-known/acme-challenge/ugFCIXCxjcnvFpPpv_cjvYBDRdCA-f8IVeEQU90VycY:
"

403 Forbidden Forbidden To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

Of course I’ve already checked a lot of things : the DNS record is good, my website is working fine on http and if I create manually the directory .well-known/acme-challenge with a single file inside, the URL is working

The server is up to date. My certbot version is 0.21.1 with apache 2.4.6. Are you aware of an issue like this with this version.

Thanks.

Christian.

[_az]

Please don’t redact the domain, it only makes it hard to help you.

Did you check that the URL also works on the IPv6/AAAA address of the domain, if it exists?

Can you provide an example URL that you believe to be working?

[christianlx]

Thanks for the reply.
One of the website is www.hotelroyalfromentin.com
I have not special IPv6 record for it.

Regards,
Christian.

[_az]

Great. Looks like it’s probably just a webroot problem.

Could you show the contents of /etc/letsencrypt/renewal/www.hotelroyalfromentin.com.conf (or whatever the closest file to that is)?

Can you also confirm that the correct webroot is reflected in that file to where the server is serving content from?

[christianlx]

I don’t have file for hotelroyalfromentin.com in this directory because I never manage to create the initial certificate certificate for it.
In this folder, I only have the files of the old websites that are working.

[_az]

Oh, okay. Sorry.

So how are you invoking Certbot?

What is the output if you try e.g. the following?

certbot certonly -a apache -d hotelroyalfromentin.com -d www.hotelroyalfromentin.com --dry-run

I noticed that PHP is serving HTTP 200 for all requests, even for non-existent paths under acme-challenge:

$ curl -i http://www.hotelroyalfromentin.com/.well-known/acme-challenge/abcdef
HTTP/1.1 200 OK
Date: Mon, 19 Feb 2018 10:35:08 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Set-Cookie: PHPSESSID=lg3696sueginlnm5ip0c64bl8o; path=/

You may need to add an exclusion for this path, if Certbot is not doing it for you, so that the request is served from the filesystem.

[christianlx]

I invock with this command : certbot --apache certonly -d hotelroyalfromentin.com -d www.hotelroyalfromentin.com

The curl command give :
curl -i http://www.hotelroyalfromentin.com/.well-known/acme-challenge/abcdef
HTTP/1.1 200 OK
Date: Mon, 19 Feb 2018 10:48:00 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips
Set-Cookie: PHPSESSID=q08p06sg1qecpd3qufanguorso; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Vary: Accept-Encoding
Content-Length: 206
Content-Type: text/html; charset=UTF-8

You mention an exclusion of this path for certbot, how can I test that ?

[_az]

I’m not sure, it depends how you have setup your virtual host. If you use .htaccess you could try put at the top:

RewriteEngine On
RewriteRule ^\.well-known - [L]

Could you show the full output of the Certbot command?

Use --dry-run at the end or else you will have rate limit problems.

[christianlx]

Here’s the htaccess, I’ve added .well-known line, reload Apache but the result is the same :

Options +FollowSymlinks
RewriteEngine On
RewriteRule ^.well-known - [L]
RewriteRule ^(uploads)/ - [L]
RewriteCond %{HTTP_HOST} !^www.
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-l
RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule ^(fr|en|es)/$ ?lng_iso=$1 [QSA]
RewriteRule ^([^/]+)/([^/]+)/$ ?lng_iso=$1&post_type=$2 [L,QSA]
RewriteRule ^([^/]+)/([^/]+)/([^/]+)$ ?lng_iso=$1&post_type=$2&slug=$3 [L,QSA]
RewriteRule ^([^/]+)/([^/]+)$ ?post_type=$1&slug=$2 [L,QSA]
RewriteRule ^([^/]+)/$ ?post_type=$1 [L,QSA]
RewriteRule ^([^/]+)/$ ?lng_iso=fr&post_type=$1 [L,QSA]

RewriteRule ^(fr|en|es)$ http://%{HTTP_HOST}/$1/ [R=301,QSA]

The copy/paste delete the backslash but the syntax is good

[christianlx]

For the moiment, I’m still stucked with this 403. Anyway, I found a strange thing in the httpd error log. Each time I try to initiate the certificate, I have the following warning :

AH00035: access to /.well-known/acme-challenge/oyENvapycgeEKyjqUtT6-_WiPEy_jQz7IXLhuAT6Ino denied (filesystem path ‘/var/lib/letsencrypt/http_challenges’) because search permissions are missing on a component of the path

Of course I gooled it and find issues with selinux or permissions. On my system, SELinux is disabled and the permissions seems to be OK :

For exemple :

drwxr-sr-x 11 apache apache 4096 Jan 30 13:43 opensadmin
-rw-r–r-- 1 apache apache 9955 Jul 17 2017 sitemap.xml
drwxr-sr-x 2 apache apache 4096 Jan 30 13:40 stats
drwxr-sr-x 5 apache apache 229376 Feb 21 15:03 uploads
drwxr-sr-x 3 apache apache 27 Feb 7 15:30 .well-known

If someone have an idea ?

[bytecamp]

because search permissions are missing on a component of the path

Show the output of the command:

namei -l /var/lib/letsencrypt/http_challenges

[christianlx]

namei -l /var/lib/letsencrypt/http_challenges/
f: /var/lib/letsencrypt/http_challenges/
dr-xr-xr-x root root /
drwxr-xr-x root root var
drwxr-xr-x root root lib
drwxr-x— root root letsencrypt
drwxrwxr-x root apache http_challenges

[sahsanu]

apache user can’t access this dir /var/lib/letsencrypt/ so you need to change the perms… or owner…or group

chmod 755 /var/lib/letsencrypt/

or

chown root:apache /var/lib/letsencrypt/

[christianlx]

Thanks a lot sahsanu. The chown command correct the problem. An outside view is always working

As it was working with the 0.20, I suspect it’s a bug in this updated RPM. I’m going to report it.

[sahsanu]

Glad you get it working but here, thanks go to @_az and @bytecamp