Certbot failed to authenticate, Invalid response

I'm having trouble renewing my domain certificate. I think it might be to do with my VHost configuration but I can't get it to work. I don't know what else to try, would appreciate any help

My domain is: www.danh.co.uk

I ran this command:

sudo certbot certonly -v -d www.danh.co.uk

It produced this output:

How would you like to authenticate with the ACME CA?


1: Apache Web Server plugin (apache)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)


Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator apache, Installer None
Certificate is due for renewal, auto-renewing...
Running pre-hook command: ufw allow http
Hook 'pre-hook' ran with output:
Rule updated
Rule updated (v6)
Renewing an existing certificate for www.danh.co.uk
Performing the following challenges:
http-01 challenge for www.danh.co.uk
Waiting for verification...
Challenge failed for domain www.danh.co.uk
http-01 challenge for www.danh.co.uk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: www.danh.co.uk
Type: unauthorized
Detail: Invalid response from http://www.danh.co.uk/.well-known/acme-challenge/7213B-1so0xhE0vCzuT_dl5er0s3VKSdfBGI_I3fNT4 [145.40.203.27]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Running post-hook command: ufw deny http
Hook 'post-hook' ran with output:
Rule updated
Rule updated (v6)
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):

Apache2

The operating system my web server runs on is (include version):

Ubuntu 20

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.22.0

Apache vhost printout:

VirtualHost configuration:
*:80 is a NameVirtualHost
default server www.danh.co.uk (/etc/apache2/sites-enabled/danh.co.uk.conf:1)
port 80 namevhost www.danh.co.uk (/etc/apache2/sites-enabled/danh.co.uk.conf:1)
alias danh.co.uk
port 80 namevhost default.only (/etc/apache2/sites-enabled/danh.co.uk.conf:40)
*:443 is a NameVirtualHost
default server 127.0.0.1 (/etc/apache2/sites-enabled/danh.co.uk.conf:61)
port 443 namevhost 127.0.0.1 (/etc/apache2/sites-enabled/danh.co.uk.conf:61)
port 443 namevhost www.danh.co.uk (/etc/apache2/sites-enabled/danh.co.uk.conf:76)
alias danh.co.uk
alias www.danh.co.uk
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex fcgid-pipe: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex fcgid-proctbl: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

Let's have a look at the file:
/etc/apache2/sites-enabled/danh.co.uk.conf

3 Likes

Please change:
ServerName www.danh.co.uk:80
to
ServerName www.danh.co.uk

There are three competing redirects:

  Redirect permanent / https://www.danh.co.uk/

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.danh.co.uk [OR]
RewriteCond %{SERVER_NAME} =danh.co.uk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

Redirect 301 /.well-known/webfinger /nextcloud/index.php/.well-known/webfinger
Redirect 301 /.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo
Redirect 301 /.well-known/carddav /nextcloud/remote.php/dav
Redirect 301 /.well-known/caldav /nextcloud/remote.php/dav

SSL is enabled on port 80:

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/www.danh.co.uk/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.danh.co.uk/privkey.pem

Please change:
ServerName www.danh.co.uk:443
to:
ServerName www.danh.co.uk

There are two competing redirects:

Redirect 301 /.well-known/webfinger /nextcloud/index.php/.well-known/webfinger
Redirect 301 /.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo
Redirect 301 /.well-known/carddav /nextcloud/remote.php/dav
Redirect 301 /.well-known/caldav /nextcloud/remote.php/dav

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule ^\.well-known/carddav /nextcloud/remote.php/dav [R=301,L]
  RewriteRule ^\.well-known/caldav /nextcloud/remote.php/dav [R=301,L]
  RewriteRule ^\.well-known/webfinger /nextcloud/index.php/.well-known/webfinger [R=301,L]
  RewriteRule ^\.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo [R=301,L]
</IfModule>
4 Likes

So I removed from the port 80:

Redirect permanent / https://www.danh.co.uk/

and

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.danh.co.uk [OR]
RewriteCond %{SERVER_NAME} =danh.co.uk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

and I removed from the port 443:

RewriteEngine on RewriteRule ^\.well-known/carddav /nextcloud/remote.php/dav [R=301,L] RewriteRule ^\.well-known/caldav /nextcloud/remote.php/dav [R=301,L] RewriteRule ^\.well-known/webfinger /nextcloud/index.php/.well-known/webfinger [R=301,L] RewriteRule ^\.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo [R=301,L]

I changed the server names to exclude the port numbers.

I restarted the apache server after making the changes but I'm still getting the same error.

I appreciate your help a lot, thank you!

Should I also remove the SSL on port 80?

1 Like

Definitely [remove that from there].

3 Likes

hmm...i've removed it and i'm still getting an invalid response

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: www.danh.co.uk
  Type:   unauthorized
  Detail: Invalid response from http://www.danh.co.uk/.well-known/acme-challenge/JujwIg2VwHewB1r2Wnd3AHUlVsIRbf5b6Z7xrr0U9cY [145.40.203.27]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

So i tried to run the standard certbot renew command and I get the following output

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/danh.co.uk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Running pre-hook command: ufw allow http
Hook 'pre-hook' ran with output:
 Rule updated
 Rule updated (v6)
Renewing an existing certificate for danh.co.uk
Performing the following challenges:
http-01 challenge for danh.co.uk
Waiting for verification...
Challenge failed for domain danh.co.uk
http-01 challenge for danh.co.uk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: danh.co.uk
  Type:   unauthorized
  Detail: Invalid response from http://danh.co.uk/.well-known/acme-challenge/luw2LDgxothm5vT0bohqEo31efBH5iydqDodjtmuJfQ [145.40.203.27]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate danh.co.uk with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.danh.co.uk-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewal configuration file /etc/letsencrypt/renewal/www.danh.co.uk-0001.conf is broken.
The error was: expected /etc/letsencrypt/live/www.danh.co.uk-0001/cert.pem to be a symlink
Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.danh.co.uk-0002.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Pre-hook command already run, skipping: ufw allow http
Renewing an existing certificate for www.danh.co.uk
Performing the following challenges:
http-01 challenge for www.danh.co.uk
Waiting for verification...
Challenge failed for domain www.danh.co.uk
http-01 challenge for www.danh.co.uk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: www.danh.co.uk
  Type:   unauthorized
  Detail: Invalid response from http://www.danh.co.uk/.well-known/acme-challenge/JmRjxb48W-EucXI-7hDkY55MWf_MUO8nSVs2yogCz58 [145.40.203.27]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate www.danh.co.uk-0002 with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.danh.co.uk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Pre-hook command already run, skipping: ufw allow http
Renewing an existing certificate for danh.co.uk and www.danh.co.uk
Performing the following challenges:
http-01 challenge for danh.co.uk
http-01 challenge for www.danh.co.uk
Waiting for verification...
Challenge failed for domain danh.co.uk
Challenge failed for domain www.danh.co.uk
http-01 challenge for danh.co.uk
http-01 challenge for www.danh.co.uk

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: danh.co.uk
  Type:   unauthorized
  Detail: Invalid response from http://danh.co.uk/.well-known/acme-challenge/tfQKRPK2D04NHRrQ_jG2XdBvAIzNmzw5eHAHYyMYijg [145.40.203.27]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

  Domain: www.danh.co.uk
  Type:   unauthorized
  Detail: Invalid response from http://www.danh.co.uk/.well-known/acme-challenge/fREN0nxfhbnlRnSPh1nnZDYI5LADbTAJsf5fSNvyUpU [145.40.203.27]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>403 Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate www.danh.co.uk with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/danh.co.uk/fullchain.pem (failure)
  /etc/letsencrypt/live/www.danh.co.uk-0002/fullchain.pem (failure)
  /etc/letsencrypt/live/www.danh.co.uk/fullchain.pem (failure)

Additionally, the following renewal configurations were invalid:
  /etc/letsencrypt/renewal/www.danh.co.uk-0001.conf (parsefail)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: ufw deny http
Hook 'post-hook' ran with output:
 Rule updated
 Rule updated (v6)
3 renew failure(s), 1 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Strangely i have several conf files for my one domain and letsencrypt is trying to renew all of them, which fail?

Let's have a look at the output of:
certbot certificates

and also:
apachectl -t -D DUMP_VHOSTS

2 Likes

so, output of certbot certificates

is:

Found the following certs:
Certificate Name: danh.co.uk
Serial Number: 4b16cc8dacb5d3e2cefa43997d3ebefe9dd
Key Type: RSA
Domains: danh.co.uk
Expiry Date: 2022-01-30 06:33:29+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/danh.co.uk/fullchain.pem
Private Key Path: /etc/letsencrypt/live/danh.co.uk/privkey.pem
Certificate Name: www.danh.co.uk-0002
Serial Number: 396f3496e30d8abdba3d1e0174bbc27c997
Key Type: RSA
Domains: www.danh.co.uk
Expiry Date: 2022-01-30 06:32:03+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/www.danh.co.uk-0002/fullchain.pem
Private Key Path: /etc/letsencrypt/live/www.danh.co.uk-0002/privkey.pem
Certificate Name: www.danh.co.uk
Serial Number: 4a3b8be94295d095566b43766be779e4452
Key Type: RSA
Domains: danh.co.uk www.danh.co.uk
Expiry Date: 2022-01-30 17:43:21+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/www.danh.co.uk/fullchain.pem
Private Key Path: /etc/letsencrypt/live/www.danh.co.uk/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/www.danh.co.uk-0001.conf

and of apachectl -t -D DUMP_V

VirtualHost configuration:
*:80 is a NameVirtualHost
default server www.danh.co.uk (/etc/apache2/sites-enabled/danh.co.uk.conf:1)
port 80 namevhost www.danh.co.uk (/etc/apache2/sites-enabled/danh.co.uk.conf:1)
alias danh.co.uk
port 80 namevhost default.only (/etc/apache2/sites-enabled/danh.co.uk.conf:40)
*:443 is a NameVirtualHost
default server default.only (/etc/apache2/sites-enabled/danh.co.uk.conf:62)
port 443 namevhost default.only (/etc/apache2/sites-enabled/danh.co.uk.conf:62)
port 443 namevhost www.danh.co.uk (/etc/apache2/sites-enabled/danh.co.uk.conf:77)
alias danh.co.uk
alias www.danh.co.uk

1 Like

That indicates a problem.
Did you try to delete any certs manually?

The first two certs are redundant with the last one:

We should have a look at this file:
/etc/apache2/sites-enabled/danh.co.uk.conf

2 Likes

No, I've not tried to delete any certificates manually. what's the procedure to that?
My site-enabled conf looks to be identified to the sites-available conf. I restarted the apache2 service after each change


<VirtualHost *:80>
        ServerAdmin me@danh.co.uk
        ServerName www.danh.co.uk
        ServerAlias danh.co.uk

        DocumentRoot /var/www/danh.co.uk/public_html

        <Directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride all
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/danherror.log
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/danhaccess.log combined

        RewriteEngine on
        RewriteCond %{SERVER_NAME} =www.danh.co.uk [OR]
        RewriteCond %{SERVER_NAME} =danh.co.uk
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>

#============================== ANTI PROXY SPAM =============================

<VirtualHost *:80>
        ServerName default.only
        <Location />
                Order allow,deny
                Deny from all
        </Location>

        ErrorLog ${APACHE_LOG_DIR}/spamerror.log
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/spamaccess.log combined

</VirtualHost>


<IfModule mod_ssl.c>

#=============================== ANTI SPAM ================================
<VirtualHost *:443>
        ServerName default.only
        <Location />
                Order allow,deny
                Deny from all
        </Location>

</VirtualHost>

#================================ WEBSITE ===================================

<VirtualHost *:443>

        ServerAdmin me@danh.co.uk
        ServerName www.danh.co.uk
        ServerAlias danh.co.uk
        Alias /nextcloud "/var/www/nextcloud"
        Alias /rainloop "/var/www/rainloop"

       Redirect 301 /.well-known/webfinger /nextcloud/index.php/.well-known/webfinger
        Redirect 301 /.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo
        Redirect 301 /.well-known/carddav /nextcloud/remote.php/dav
        Redirect 301 /.well-known/caldav /nextcloud/remote.php/dav

        SSLEngine on
        SSLCertificateFile /etc/letsencrypt/live/www.danh.co.uk/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/www.danh.co.uk/privkey.pem

        DocumentRoot /var/www/danh.co.uk/public_html
        <Directory /var/www/danh.co.uk/public_html>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride all
                Order allow,deny
                allow from all

       </Directory>

        <Directory /var/www/nextcloud/>
                Require all granted
                AllowOverride All
                Options FollowSymLinks MultiViews

                <IfModule mod_dav.c>
 Dav off
                </IfModule>
        </Directory>

        <Directory /var/www/rainloop>
                DirectoryIndex index.php
                Options -Indexes +ExecCGI
                AllowOverride All
                Order deny,allow
                Allow from all
                Require all granted
        </Directory>

        <Directory /var/www/rainloop/data>
                Options -Indexes
                Deny from all
        </Directory>


        <IfModule mod_headers.c>
                Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
        </IfModule>

        ErrorLog ${APACHE_LOG_DIR}/danherror.log
        LogLevel warn
        CustomLog ${APACHE_LOG_DIR}/danhssl_access.log combined

        SSLEngine on

        <FilesMatch "\.(cgi|shtml|phtml|php)$">
                SSLOptions +StdEnvVars
  </FilesMatch>

        <Directory /usr/lib/cgi-bin>
                SSLOptions +StdEnvVars
        </Directory>

        BrowserMatch "MSIE [2-6]"nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
        # MSIE 7 and newer should be able to use keepalive
        BrowserMatch "MSIE [17-9]"ssl-unclean-shutdown
        ServerAlias www.danh.co.uk
        Include /etc/letsencrypt/options-ssl-apache.conf
        SSLCertificateFile /etc/letsencrypt/live/www.danh.co.uk/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/www.danh.co.uk/privkey.pem
</VirtualHost>

</IfModule>

Ignoring the duplicative certs for a moment. Right now I get a 403 error like you see in your first post. Do you have a deny statement at a higher level in your Apache conf files?

I ask because your Directory statement in the VirtualHost will not be effective when using the --apache plug-in which you are using. That adds temporary changes to your VirtualHost and does not rely on the DocumentRoot folder. So, I think (not 100% sure) that this is why it is failing.

Example

curl -i www.danh.co.uk/.well-known/acme-challenge/ForumTest123

HTTP/1.1 403 Forbidden
Date: Thu, 03 Feb 2022 22:03:25 GMT
Server: Apache
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
2 Likes

There is a likely name:port overlap

Also, this seems redundant:

2 Likes

This looks like one to check. I should be looking in the apache2.conf and the ports.conf files, is that right?

couldn't figure out why the challenge location kept failing. I even added exceptions for .well-known.
So I instead switched my ddns to desec.io and authenticated using their certbot plugin.
Thanks for trying to work it out with me!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

A post was split to a new topic: Needs to be merged with existing [hard to find] topic