Renewal failed on Apache on http-01 challenge returned with 403

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: soulsurfer.ddns.net

I ran this command: certbot -v

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Certificate is due for renewal, auto-renewing...
Renewing an existing certificate for soulsurfer.ddns.net
Performing the following challenges:
http-01 challenge for soulsurfer.ddns.net
Waiting for verification...
Challenge failed for domain soulsurfer.ddns.net
http-01 challenge for soulsurfer.ddns.net

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: soulsurfer.ddns.net
Type: unauthorized
Detail: 87.123.125.228: Invalid response from http://soulsurfer.ddns.net/.well-known/acme-challenge/xZl9E4pGJVUSVNSb36cPtZU0fyMK0gyle_ERI4Nqh9I: 403

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges

My web server is (include version): Apache 2.4.52

The operating system my web server runs on is (include version): Ubuntu 22.04.4 LTS - jammy

My hosting provider, if applicable, is: selfhosted

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.21.0

I had no problems for years, but I found that some Ubuntu update blocked http on port 80.
having fixed this, I can access the .well-known/... files via http:

curl http://soulsurfer.ddns.net/.well-known/acme-challenge/letsdebug-test -v

GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1
Host: soulsurfer.ddns.net
User-Agent: curl/8.2.1
Accept: /

< HTTP/1.1 200 OK
< Date: Thu, 25 Apr 2024 20:22:30 GMT
< Server: Apache/2.4.52
< X-Content-Type-Options: nosniff
< X-Frame-Options: sameorigin
< Strict-Transport-Security: max-age=15552000;includeSubdomains
< X-XSS-Protection: 1; mode=block
< Referrer-Policy: strict-origin-when-cross-origin
< X-Robots-Tag: “none”
< X-Download-Options: “noopen”
< X-Permitted-Cross-Domain-Policies: “none”
< Last-Modified: Thu, 25 Apr 2024 20:21:14 GMT
< ETag: "49-616f1875f2738"
< Accept-Ranges: bytes
< Content-Length: 73
<

HelloHello, world.

The status code for missing files is 404, as expected. But certbot gets 403 for the http-01 renewal check.
I have a nextcloud server, version 28, and php-8.1 installed since 2016.

The apache log states the forbidden access:
AH01797: client denied by server configuration: /var/lib/letsencrypt/http_challenges/8fUCnNLytzOOpYCU7xV2Sm5jpljPMXnO0FyK7qgc-Gg

The file accesses are set like this:

ls -l /var/lib/letsencrypt/http_challenges/
drwxrwxr-x 2 root www-data 4096 Apr 25 23:32 ./
drwxrwxr-x 4 root www-data 4096 Apr 25 23:32 ../

I'm grateful for any hints ...

2 Likes
2

It is not certbot that ever tries to reach your site - it is LE.
So that should read: "LE gets 403"
That said, I think you're reading too much into that 403 response.
I don't think it means that LE heard a 403 response from your system.
[the complete logs would probably make things clearer]

Anywho...
Since we are dealing with Apache, I would suggest that we begin at the very beginning:
Let's have a look at the output of:
sudo apachectl -t -D DUMP_VHOSTS

2 Likes
3

Hi, rg305,

I'm confused . My cronjob ran successfully last night,
and I have a new certificate - valid since Fri, 26 Apr 2024 03:48:36 GMT

I surely started certbot as root, and it failed always.
Even with the latest static change I made, when I tried to check, what's happing while certbot modified the apache configuration and use the http_challange directory temporarily. I made the change myself for testing, but I got only 404 instead of 403 even for existing files.

  <Directory /var/lib/letsencrypt/http_challenges>
    Require all granted
  </Directory>
  <Location /.well-known/acme-challenge>
    Require all granted
  </Location>

I restarted apache2, changed access right to /var/lib/letsencrypt/, but nothing worked.
So, I have no clue at all, why it failed constantly for days and why THIS NIGHT it DID work :face_with_spiral_eyes:.

But here's the vhost dump you requested:

VirtualHost configuration:
*:80                   soulsurfer.ddns.net (/etc/apache2/sites-enabled/nextcloud.conf:3)
*:443                  soulsurfer.ddns.net (/etc/apache2/sites-enabled/nextcloud.conf:33)

And this is the nextcloud.conf that worked for years with certbot :

<VirtualHost *:80>
  DocumentRoot /var/www/html
  ServerName soulsurfer.ddns.net

  <Directory />
    Order Deny,Allow
    Deny from All
  </Directory>

  <Directory /var/www/html>
     Options Indexes FollowSymLinks MultiViews
     AllowOverride None
     Order allow,deny
     allow from all
  </Directory>

#  RewriteEngine off
#  ReWriteCond %{SERVER_PORT} !^443
#  RewriteRule ^/(.*) https://%{ServerName}/$1 [NC,R,L]
#  RewriteCond %{SERVER_NAME} =soulsurfer.ddns.net
#  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
<VirtualHost *:443>
   ServerName soulsurfer.ddns.net
   ProtocolsHonorOrder On
   Protocols h2 http/1.1
   # currently h2 is not working due to php
   SSLEngine On
   SSLCertificateFile      /etc/letsencrypt/live/soulsurfer.ddns.net/cert.pem
   SSLCertificateKeyFile   /etc/letsencrypt/live/soulsurfer.ddns.net/privkey.pem
   SSLCertificateChainFile /etc/letsencrypt/live/soulsurfer.ddns.net/chain.pem

   DocumentRoot /var/www/html

   <IfModule mod_jk.c>
     # ... some tomcat9 managed webapps here
     # ... had not problems before
   </IfModule>

   <Directory /var/www/html/>
     Options +FollowSymLinks
     AllowOverride All
     order allow,deny
     allow from all
   </Directory>
</VirtualHost>

best regards and thanks for "standing by" (that did the trick, I guess :wink: )

2 Likes
4

Requests to:
soulsurfer.ddns.net/.well-known/acme-challenge/
would be served from:

/var/www/html/.well-known/acme-challenge/
NOT from:
/var/lib/letsencrypt/http_challenges

Perhaps something is blocking access to the current location.

1 Like
5

@Soulsurfering here is a list of issued certificates crt.sh | soulsurfer.ddns.net, the latest being 2024-04-26.
Presently being served certificate https://decoder.link/sslchecker/soulsurfer.ddns.net/443 looks to be
crt.sh | 12856425320

Is this issue resolved?

1 Like
6

Hi Bruce5051,

yes, some how the error just went somewhere else. I stated that already, but obviously I added too much text afterwards. Sorry.

Actually I still like to understand, what went wrong and what solved the failure.
I can't checkmark 'solution' box, because I haven't found one ...

best regards

2 Likes