Apache ProxyPassReverse: Certbot returns Invalid response (403) when trying to renew or getting new certificates

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
myaccounts.live

I ran this command:
$ sudo certbot certonly --apache --dry-run --test-cert --renew-by-default --agree-tos -d web.myaccounts.live -d webmin.myaccounts.live -d myaccounts.live

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for myaccounts.live
http-01 challenge for web.myaccounts.live
http-01 challenge for webmin.myaccounts.live
Waiting for verification...
Challenge failed for domain myaccounts.live
Challenge failed for domain web.myaccounts.live
Challenge failed for domain webmin.myaccounts.live
http-01 challenge for myaccounts.live
http-01 challenge for web.myaccounts.live
http-01 challenge for webmin.myaccounts.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: myaccounts.live
   Type:   unauthorized
   Detail: 162.250.127.130: Invalid response from
   http://myaccounts.live/.well-known/acme-challenge/dD5TV7HwZsytQbiIlax3Y3osbX1VWcXit7Jpmu0NxWo:
   403

   Domain: web.myaccounts.live
   Type:   unauthorized
   Detail: 162.250.127.130: Invalid response from
   http://web.myaccounts.live/.well-known/acme-challenge/04azeGlIT7nvy7jaa9k7huDFn6XNmidZHloKd9J4u_4:
   403

   Domain: webmin.myaccounts.live
   Type:   unauthorized
   Detail: 162.250.127.130: Invalid response from
   http://webmin.myaccounts.live/.well-known/acme-challenge/J8ChUMv6-54Gr8XDcKAQt1K_umlInZ1vBhdpqZT3-Mo:
   403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):

$ apachectl -v
Server version: Apache/2.4.41 (Ubuntu)
Server built:   2022-03-16T16:52:53

The operating system my web server runs on is (include version):

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

My hosting provider, if applicable, is:
InterServer

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No control panel, only command line

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

$ certbot --version
certbot 0.40.0

I'm using the Apache for the Proxy Reverse only, and the enabled sites conf contents are seperated in three conf's:
First file:

<IfModule mod_ssl.c>
<VirtualHost *:443>
	ProxyPreserveHost On
	ProxyRequests Off
	ServerName myaccounts.live
	ProxyPass / http://localhost:8080/
	ProxyPassReverse / http://localhost:8080/
	<Proxy *>
		Order deny,allow
		Allow from all
	</Proxy>

	SSLCertificateFile /etc/letsencrypt/live/myaccounts.live/fullchain.pem
	SSLCertificateKeyFile /etc/letsencrypt/live/myaccounts.live/privkey.pem
	Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

<VirtualHost *:80>
        ServerName myaccounts.live
        Redirect Permanent / https://myaccounts.live/
        RewriteEngine on
        RewriteCond %{SERVER_NAME} =myaccounts.live
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

</IfModule>

Second file:

<IfModule mod_ssl.c>

<VirtualHost *:443>
        ProxyPreserveHost On
        ProxyRequests Off
        ServerName webmin.myaccounts.live
        ProxyPass / http://localhost:4558/
        ProxyPassReverse / http://localhost:4558/

        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>

	Include /etc/letsencrypt/options-ssl-apache.conf
	SSLCertificateFile /etc/letsencrypt/live/webmin.myaccounts.live/fullchain.pem
	SSLCertificateKeyFile /etc/letsencrypt/live/webmin.myaccounts.live/privkey.pem
</VirtualHost>

<VirtualHost *:80>
        ServerName 'webmin.myaccounts.live'
	Redirect Permanent / http://webmin.myaccounts.live/
</VirtualHost>

</IfModule>

Third file:

<VirtualHost *:80>
        ServerName web.myaccounts.live
	DocumentRoot /var/www/html
	ProxyPreserveHost On 
        ProxyPass / http://localhost:4558/
        ProxyPassReverse / http://localhost:4558/
</VirtualHost>

As you can see, there are old certs installed in the first and second files (myaccounts.live and webmin.myaccounts.live) and I'm trying to test renewing them and getting a new cert for the third subdomain.

Hi @SJaafar, and welcome to the LE community forum :slight_smile:

In the first file, I see two redirections and nothing configured specifically to handle the challenge files:

Not that it is required...
But it would seem then certbot 0.40.0 is unable to properly handle the --apache authentication.
[look into updating certbot to a newer version]
I'd recommend switching to --webroot authentication and redo the port 80 server blocks.
To something more like:

<VirtualHost *:80>
  ServerName myaccounts.live

  #this folder must be created and changed to fit your need
  DocumentRoot /some/designated/challenge/root

  #skip challenge requests and
  <LocationMatch "^/(?!\.well-known)">
    #send all other requests to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
  </LocationMatch>
</VirtualHost *:80>
2 Likes

Also look into the possible IPv6 issue:

Name:      myaccounts.live
Addresses: 2604:a00:12:2169:5054:ff:224:69
           162.250.127.130
2 Likes

Another odd thing ... Each of the challenge URLs in first post were to HTTP. But, the port 80 server for myaccounts.live redirects all requests to HTTPS. Why did the redirect not happen?

And, the redirect here goes back to itself?

2 Likes

Thanks for the reply.

I've edited the first conf file to be:

<VirtualHost *:80>
        ServerName myaccounts.live
        DocumentRoot /var/www/html
        #skip challenge requests and
        <LocationMatch "^/(?!\.well-known)">
                #send all other requests to HTTPS
                RewriteEngine On
                RewriteCond %{HTTPS} !=on
                RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
        </LocationMatch>
</VirtualHost>

And ran the commands:

$ sudo systemctl stop apache2
$ sudo certbot certonly --webroot --dry-run --test-cert --renew-by-default --agree-tos -d myaccounts.live

But I've got this result:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for myaccounts.live
Input the webroot for myaccounts.live: (Enter 'c' to cancel): /var/www/html/
Waiting for verification...
Challenge failed for domain myaccounts.live
http-01 challenge for myaccounts.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: myaccounts.live
   Type:   connection
   Detail: 162.250.127.130: Fetching
   http://myaccounts.live/.well-known/acme-challenge/2BjHucgDr0gwqjJ_U_VjLV8_p613Ec6wZ1_V-zxdClo:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

And you can see there is a folder for the webroot:

$ ls -al /var/www/html/
total 20
drwxr-xr-x 2 www-data www-data  4096 May 12 07:46 .
drwxr-xr-x 3 root     root      4096 May  1 22:04 ..
-rw-r--r-- 1 www-data www-data 10918 May  1 22:04 index.html

Thanks for the reply.

I'm trying to setup a new url to redirect all http to the expected new https of the same url, and the https doesn't have a cert yet.
I dont' need to have them?

1 Like

Like what?

This combination is incorrect.

The --webroot method requires that there is an existing web server running (on port 80) at the time that the certificate is requested. (It uses the existing web server to complete the CA's challenges.)

The --standalone method requires that there is not an existing web server running (on port 80) at the time that the certificate is requested. (It creates its own temporary substitute web service.)

If you consider --webroot to be the most useful method for your situation (which it may be), you must not stop your web server before running Certbot!

2 Likes

Oh, my bad.

Now the same command run and slightly another error was returned:

$ sudo certbot certonly --webroot --dry-run --test-cert --renew-by-default --agree-tos -d myaccounts.live
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for myaccounts.live
Input the webroot for myaccounts.live: (Enter 'c' to cancel): /var/www/html/
Waiting for verification...
Challenge failed for domain myaccounts.live
http-01 challenge for myaccounts.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: myaccounts.live
   Type:   unauthorized
   Detail: 2604:a00:12:2169:5054:ff:224:69: Invalid response from
   http://myaccounts.live/.well-known/acme-challenge/ANQBRG20IA6lUzdO9gOP8wN5YAxFUMCZMJr6QA7mJxE:
   403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Also,

is the obsolete name for --force-renewal, which is not relevant or useful here. --dry-run already forces the test to attempt to issue a test certificate. The --force-renewal option would only be relevant if there were an existing certificate that you were attempting to renew before Certbot thought it was time to do so.

Unfortunately, the by-default part here was not meant to meant "automatically (in the future)"; Certbot always tries to renew certificates automatically in the future. It was meant to mean "(right now) without asking me". This phrasing was confusing, which is why the option name was changed to --force-renew and doesn't appear in recent documentation.

2 Likes

When I go to your site, I see a 403 error even for the home page, not just for the Let's Encrypt challenge file. (I don't see the content of the index.html page that you showed exists.) So, your current web site configuration is not allowing the public to download content from your site at all—including web page content but also including Let's Encrypt challenge files.

1 Like

Sorry, I guess that is much the same error that you were originally getting that led you to start this thread. :frowning: I'm sorry, but I don't know offhand why your configuration prevents visitors from accessing any site content. Maybe look in the Apache error log to see if it logged a more detailed explanation?

1 Like

Yes, the main website was meant to use the proxy to be redirected to another port on the same url as you can see in the first conf file of the main post.

The Apache is used here mainly for the LE certs creating/renewal and redirecting to the port 443 of the domain and subdomains which has other mini servers.

Like: LE will prefer IPv6 over IPv4 when available.

If IPv6:80 can't reach your web server... then all will fail.

2 Likes

Maybe something related to the ProxyReverse in the main page?

$ ls -al /var/www/html/.well-known/acme-challenge/test.txt
-rw-r--r-- 1 www-data www-data 11 May 12 13:48 /var/www/html/.well-known/acme-challenge/test.txt

$ curl http://myaccounts.live/.well-known/acme-challenge/test.txt
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://myaccounts.live/.well-known/acme-challenge/test.txt">here</a>.</p>
</body></html>

But the main website http://myaccounts.live:80 can be accessed from the browser and it's been redirected to the port 443 which is the expected action.

$ curl http://myaccounts.live:80
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://myaccounts.live/">here</a>.</p>
</body></html>

In the system that operates "the browser", does it support IPv6?
Do you understand the difference between IPv6 and IPv4?

I get this on IPv6:

curl -Ii6 http://myaccounts.live/.well-known/acme-challenge/test.txt
HTTP/1.1 403 Forbidden
Connection: close
Content-Type: text/html
Cache-Control: no-cache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Security-Policy: frame-ancestors 'self'
Content-Length: 3482

and this on IPv4:

curl -Ii4 http://myaccounts.live/.well-known/acme-challenge/test.txt
HTTP/1.1 301 Moved Permanently
Date: Thu, 12 May 2022 22:00:14 GMT
Server: Apache
Location: https://myaccounts.live/.well-known/acme-challenge/test.txt
Content-Type: text/html; charset=iso-8859-1
2 Likes

I've disabled ufw and fail2ban but that didn't help. :confused:

I've run the curl command on the same server and I got the same output of the two options:

$ curl -Ii6 http://myaccounts.live/.well-known/acme-challenge/test.txt
HTTP/1.1 301 Moved Permanently
Date: Sun, 15 May 2022 05:14:15 GMT
Server: Apache
Location: https://myaccounts.live/.well-known/acme-challenge/test.txt
Content-Type: text/html; charset=iso-8859-1

$ curl -Ii4 http://myaccounts.live/.well-known/acme-challenge/test.txt
HTTP/1.1 301 Moved Permanently
Date: Sun, 15 May 2022 05:14:29 GMT
Server: Apache
Location: https://myaccounts.live/.well-known/acme-challenge/test.txt
Content-Type: text/html; charset=iso-8859-1

So, it's been related to the firewall?

That 301 redirect is totally fine, but the trouble is what happens next (on the HTTPS service):

$ curl https://myaccounts.live/.well-known/acme-challenge/test.txt
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

Why is the HTTPS service unwilling to serve the file?

2 Likes