From TLS-SNI-01 to HTTP-01: 403 (Forbidden) - solved

[martinschniewind]

My domain is: https://wolkepur.de

I ran this command: certbot renew

It produced this output: n.a.

My web server is (include version): apache2

The operating system my web server runs on is (include version): ubuntu 14.04

My hosting provider, if applicable, is: n.a.

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.28

Spent some days, and now I'm here for sharing the solution:

In short:
chmod 755 /var/lib/letsencrypt
did the trick.

The long story:

After having received the "Action Required" e-mail, I upgraded certbot from ppa:certbot/certbot as recommended.

But, renewal failed: the letsencrypt verification server was not permittet to access "/.well-known/acme-challenge/the-file-name-of-the challenge", whatever vhost included in my certificate i tried. My apache answered "403 forbidden". The client reported:

Failed authorization procedure
The client lacks sufficient authorization
To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

First, I suspected certbot not to create the challenges and the .well-known folders as expected and fiddled around with creating them manually. Without success...

Then I ran certbot with
certbot --debug-challenges -v
and examined its output.

There I found that certbot temporarily changes the configuration for each enabled non-ssl site in /etc/apache2/sites-enabled by adding two includes (and removing them on termination). These includes make apache redirect all accesses to "/.well-known/acme-challenge" to "/var/lib/letseincrypt/http_challenges". That ist the place where certbot creates the challenges (temporarily, too - they will bi removed after use).

This was tricky, because this is not mentioned in these parts of documentation I read; neither it had been mentioned by any other people struggling around with 403-errors.

A brief look at /var/log/apache2/error.log revealed that access was denied

because search permissions are missing on a component of the path

Indeed, /var/lib/letsencrypt did not have execution rights for others, what means that others are not allowed to browse this directory. Adding this fixed the problem; apache was able to deliver contents of /var/lib/letsencrypt/http_challenges.

[mnordhoff]

@joohoi

In this thread, a second person had issues with /var/lib/letsencrypt permissions.

@martinschniewind

Do you remember exactly what the permissions/owner of the directory were before you fixed it?

Has anything ever happened that might have messed up that directory? Like restoring from a backup, or copying the directory from a different system?

[martinschniewind]

Permissions have been 754 or 750 - I’m sorry, I don’t remember exactly.

Beside this server I am maintaining another two, which run Ubuntu 16.04. All of them use letsencrypt certificates, but they have been set up at different times. So yesterday I compared their histories.

This is remarkable: The most recent one had originally been set up with certbot-auto, and permissions of /var/lib/letsencrypt were as expected.

Both of the other ones started encryption with ubuntu’s package ‘python-letsencrypt-apache’ and had wrong permissions on this directory. So I presume that either the package’s scripts or let’s encrypts auto itself created /var/lib/letsencrypt.

Maybe certbot should check permissions before executing renewals. There could be a lot of cases like this, if I’m right…

[joohoi]

Thanks for taking time to look into this @martinschniewind !

We’re looking into ways to fix this runtime!