Apache ProxyPassReverse: Certbot returns Invalid response (403) when trying to renew or getting new certificates

I have a reverse proxy to another mini server on the port (8080).
This might be the cause?

I see, I looked a little more at what you wrote above. Sorry for not checking over that carefully.

Are you sure that your LocationMatch is doing what you want here? In particular, it seems that the /.well-known/acme-challenge request is getting redirected to HTTPS, which suggests that the LocationMatch did not succeed in matching it and thereby causing it to get treated specially.

Can you check the Apache documentation for LocationMatch to be sure that the exception that you've written is correct and will do what you expect?

2 Likes

It is likely only doing IPv4.
What is doing IPv6?
[which gives a similar - but different response]

curl -Ii4 http://myaccounts.live/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 403 Forbidden
Date: Sun, 15 May 2022 06:14:52 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

curl -Ii6 http://myaccounts.live/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 403 Forbidden
Connection: close
Content-Type: text/html
Cache-Control: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: frame-ancestors 'self'
Content-Length: 3488
2 Likes

And since it's Apache...
Have you verified that there are no name:port overlaps?
Show the output of:
apachectl -t -D DUMP_VHOSTS

2 Likes

I've edited the conf file before (post# 5) and retired it again as per your suggestion:

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ProxyPreserveHost On
        ProxyRequests Off
        ServerName myaccounts.live
        ProxyPass /.well-known !
        ProxyPass / http://localhost:8080/
        ProxyPassReverse / http://localhost:8080/
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>

        SSLCertificateFile /etc/letsencrypt/live/myaccounts.live/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/myaccounts.live/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

<VirtualHost *:80>
        ServerName myaccounts.live
        DocumentRoot /var/www/html
        #skip challenge requests and
        <LocationMatch "^/(?!\.well-known)">
                #send all other requests to HTTPS
                RewriteEngine On
                RewriteCond %{HTTPS} !=on
                RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
        </LocationMatch>
</VirtualHost>
</IfModule>

and here is the output:

$ sudo systemctl reload apache2
$ sudo certbot certonly --apache --dry-run --test-cert --agree-tos -d web.myaccounts.live
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for web.myaccounts.live
Waiting for verification...
Challenge failed for domain web.myaccounts.live
http-01 challenge for web.myaccounts.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: web.myaccounts.live
   Type:   unauthorized
   Detail: 2604:a00:12:2169:5054:ff:224:69: Invalid response from
   http://web.myaccounts.live/.well-known/acme-challenge/nsIVHW7_q094CjDc9s4AYhUpUtNdY_1qv-naXUSynCE:
   403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Here we go:

$ sudo apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server myaccounts.live (/etc/apache2/sites-enabled/myaccounts-le-ssl.conf:2)
         port 443 namevhost myaccounts.live (/etc/apache2/sites-enabled/myaccounts-le-ssl.conf:2)
         port 443 namevhost webmin.myaccounts.live (/etc/apache2/sites-enabled/webmin.conf:3)
*:80                   is a NameVirtualHost
         default server myaccounts.live (/etc/apache2/sites-enabled/myaccounts-le-ssl.conf:19)
         port 80 namevhost myaccounts.live (/etc/apache2/sites-enabled/myaccounts-le-ssl.conf:19)
         port 80 namevhost webmin.myaccounts.live (/etc/apache2/sites-enabled/webmin.conf:27)
         port 80 namevhost web.myaccounts.live (/etc/apache2/sites-enabled/webmin_ver2.conf:1)

Can you remove the AAAA record from DNS?
[just as a test]

1 Like

That section lacks a DocumentRoot.
Where would the /.well-known matched requests go?

It should match the HTTP server block:

1 Like

Why are you showing the server blocks for "myaccounts.live" and then testing "web.myaccounts.live" ?

You don't need to say both of these:
--dry-run --test-cert
[Pick one]

1 Like

If you want to troubleshoot "web", then show this file:

1 Like

Sorry, I expected the error is caused by some redirects of the main page not the subdomain.

Here is the conf :

<VirtualHost *:80>
        ServerName web.myaccounts.live
        ProxyRequests Off
        ProxyPreserveHost On
        ProxyPass / http://myaccounts.live:4558/
        ProxyPassReverse / http://myaccounts.live:4558/
</VirtualHost>

The output is:

$ sudo certbot certonly --apache --dry-run --agree-tos -d web.myaccounts.live
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for web.myaccounts.live
Waiting for verification...
Challenge failed for domain web.myaccounts.live
http-01 challenge for web.myaccounts.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: web.myaccounts.live
   Type:   unauthorized
   Detail: 2604:a00:12:2169:5054:ff:224:69: Invalid response from
   http://web.myaccounts.live/.well-known/acme-challenge/3CeHjNzvQ0PqLptYAn2yPXAmIwtY_T16PrSkRFgv-ic:
   403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I've removed the AAAA records and here is the output:

$ sudo certbot certonly --apache --dry-run --agree-tos -d web.myaccounts.live
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for web.myaccounts.live
Waiting for verification...
Challenge failed for domain web.myaccounts.live
http-01 challenge for web.myaccounts.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: web.myaccounts.live
   Type:   unauthorized
   Detail: 162.250.127.130: Invalid response from
   http://web.myaccounts.live/.well-known/acme-challenge/5NFbjT-MBD4GVkeIErB42rYzLFoLLlr5v6aKRi3MmM8:
   403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Well removing AAAA did not change the error - still 403

I see for "web" there is no handling for the challenge requests:

Like there is for:

1 Like

Finally, I've got it worked.

$ sudo certbot certonly --apache --dry-run --agree-tos -d myaccounts.live
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for myaccounts.live
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - The dry run was successful.

In /etc/apache2/apache2.conf, there are two lines I needed to comment:

<Directory />
        Options FollowSymLinks
        AllowOverride None
        Order deny,allow
#       Deny from all
#       Require all denied
</Directory>

Thanks, all of you for your helpful support.
Have a nice day.

1 Like