Cannot Renew Certificate - authorisation error

Metricrat · May 11, 2020, 10:42am

My domain is: ai2.metricrat.co.uk (virtual host)
OS/Webserver: Ubuntu 18.04LTS Apache
Server: IONOS VPS, can login as root to shell
A Record: @ 77.68.114.189 (for metricrat.co.uk)
CNAME: ai2 ghs.googlehosted (hosting a new google site)
Cerbot Version: certbot 0.31.0

Virtual Host Info:

/etc/apache2/sites-available/ai2.metricrat.co.uk.conf:

<VirtualHost *:80>
         ServerAdmin webmaster@ai2.metricrat.co.uk
         ServerName ai2.metricrat.co.uk
         DocumentRoot /var/www/ai2.metricrat.co.uk/public_html
         ErrorLog ${APACHE_LOG_DIR}/error.log
         CustomLog ${APACHE_LOG_DIR}/access.log combined
     RewriteEngine on
     RewriteCond %{SERVER_NAME} =ai2.metricrat.co.uk
     RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
     </VirtualHost>

ls -al /etc/letsencrypt/live/ai2.metricrat.co.uk:

total 12
     drwxr-xr-x 2 root root 4096 May 10 23:04 .
     drwx------ 9 root root 4096 Mar  2 17:18 ..
     lrwxrwxrwx 1 root root   43 Mar  1 23:08 cert.pem -> ../../archive/ai2.metricrat.co.uk/cert1.pem
     lrwxrwxrwx 1 root root   44 Mar  1 23:08 chain.pem -> ../../archive/ai2.metricrat.co.uk/chain1.pem
     lrwxrwxrwx 1 root root   48 Mar  1 23:08 fullchain.pem -> ../../archive/ai2.metricrat.co.uk/fullchain1.pem
     lrwxrwxrwx 1 root root   46 Mar  1 23:08 privkey.pem -> ../../archive/ai2.metricrat.co.uk/privkey1.pem
     -rw-r--r-- 1 root root  692 Mar  1 23:08 README

/etc/letsencrypt/renewal/ai2.metricrat.co.uk.conf:

#renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/ai2.metricrat.co.uk
cert = /etc/letsencrypt/live/ai2.metricrat.co.uk/cert.pem
privkey = /etc/letsencrypt/live/ai2.metricrat.co.uk/privkey.pem
chain = /etc/letsencrypt/live/ai2.metricrat.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/ai2.metricrat.co.uk/fullchain.pem
    
# Options used in the renewal process
[renewalparams]
account = f8ca3af888a38836c998c68cb9bfd44e
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

Dry Run Renewal Output:

 sudo certbot renew --dry-run
 
 Processing /etc/letsencrypt/renewal/ai2.metricrat.co.uk.conf
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Cert is due for renewal, auto-renewing...
 Plugins selected: Authenticator apache, Installer apache
 Renewing an existing certificate
 Performing the following challenges:
 http-01 challenge for ai2.metricrat.co.uk
 Waiting for verification...
 Cleaning up challenges
 Attempting to renew cert (ai2.metricrat.co.uk) from /etc/letsencrypt/renewal/ai2.metricrat.co.uk.conf produced an unexpected error: Failed authorization procedure. ai2.metricrat.co.uk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ai2.metricrat.co.uk/.well-known/acme-challenge/n4OB7hFuMaMb9600YmwJI5aD4OEjLjS-V4RYxvA0PAA [2607:f8b0:400f:801::2013]: "<!DOCTYPE html><html lang=\"en-US\" itemscope itemtype=\"http://schema.org/WebPage\"><head><script type=\"text/javascript\" nonce=\"uvE". Skipping.

Also tried with webroot:

sudo certbot certonly -d ai2.metricrat.co.uk
.....
Failed authorization procedure. ai2.metricrat.co.uk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ai2.metricrat.co.uk/.well-known/acme-challenge/9-cbm2GPnZOdQcujZ5TK8eI229mp7zJ9Ig2n1M2Ytss [2607:f8b0:400f:800::2013]: "<!DOCTYPE html><html lang=\"en-US\" itemscope itemtype=\"http://schema.org/WebPage\"><head><script type=\"text/javascript\" nonce=\"QW8"

I have five other virtual hosts (main domains) that either renew OK or are reporting OK if not yet renewable.

What can I try next?
What other info may be needed to resolve?
I have 18 days left!
Thanks in anticipation

rg305 · May 11, 2020, 10:47am

Change #1:
The HTTP auth requests are being redirected to HTTPS.
[all requests are redirected]
I would handle them in HTTP.

rg305 · May 11, 2020, 10:50am

Please show this file contents:
/etc/letsencrypt/renewal/ai2.metricrat.co.uk.conf

Metricrat · May 11, 2020, 10:53am

I did provide this above (there is a lot of stuff!)

Metricrat:

/etc/letsencrypt/renewal/ai2.metricrat.co.uk.conf:

#renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/ai2.metricrat.co.uk
cert = /etc/letsencrypt/live/ai2.metricrat.co.uk/cert.pem
privkey = /etc/letsencrypt/live/ai2.metricrat.co.uk/privkey.pem
chain = /etc/letsencrypt/live/ai2.metricrat.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/ai2.metricrat.co.uk/fullchain.pem
    
# Options used in the renewal process
[renewalparams]
account = f8ca3af888a38836c998c68cb9bfd44e
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

Metricrat · May 11, 2020, 10:53am

OK, how do I go about doing that ?

rg305 · May 11, 2020, 10:55am

Sorry missed it.
Nothing really out-of-the-ordinary there

rg305 · May 11, 2020, 10:58am

Depends on the web server.
In Apache, you can use an Alias directive:
Alias /.well-known/acme-challenge/ /some/local/folder/
[insert that like below DocumentRoot line]
[folder may need Directory section to difine/allow access to it]

Metricrat · May 11, 2020, 11:12am

Like this?

<VirtualHost *:80>
         ServerAdmin webmaster@ai2.metricrat.co.uk
         ServerName ai2.metricrat.co.uk
         DocumentRoot /var/www/ai2.metricrat.co.uk/public_html

         Alias /.well-known/acme-challenge/ /home/metricrat/cert/

         ErrorLog ${APACHE_LOG_DIR}/error.log
         CustomLog ${APACHE_LOG_DIR}/access.log combined
     RewriteEngine on
     RewriteCond %{SERVER_NAME} =ai2.metricrat.co.uk
     RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
     </VirtualHost>

rg305 · May 11, 2020, 11:15am

OK time for a simple test.
Place a test-file in that folder.
like:
echo "test" >> /home/metricrat/cert/test-file

[after Apache has been restarted to take that change]
Let’s try accessing that file via the Internet:
http://ai2.metricrat.co.uk/.well-known/acme-challenge/test-file

If good, then run certbot command again.
If no good, then we need to troubleshoot that…
[fingers crossed]

rg305 · May 11, 2020, 11:21am

We may need to modify the Alias line into a location+alias:

<location "/.well-known/acme-challenge/">
   Alias "/home/metricrat/cert/"
</location>

Metricrat · May 11, 2020, 11:42am

Tried both methods,

...DocumentRoot /var/www/ai2.metricrat.co.uk/public_html

    Alias /.well-known/acme-challenge/ /home/metricrat/cert/
    
    ErrorLog ${APACHE_LOG_DIR}/error.log...

and

   ... DocumentRoot /var/www/ai2.metricrat.co.uk/public_html

        <location "/.well-known/acme-challenge/">
           Alias "/home/metricrat/cert/"
        </location>

        ErrorLog ${APACHE_LOG_DIR}/error.log...

both take me to a 404 page on my google site:

Not able to got to http://ai2.metricrat.co.uk, browser just redirects to https://ai2.metricrat.co.uk

rg305 · May 11, 2020, 11:45am

Try:

<VirtualHost *:80>
    ServerAdmin webmaster@ai2.metricrat.co.uk
    ServerName ai2.metricrat.co.uk
    DocumentRoot /var/www/ai2.metricrat.co.uk/public_html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    <location "/.well-known/acme-challenge/">
         Alias "/home/metricrat/cert/"
    </location>
    <location "/">
         Redirect / https://ai2.metricrat.co.uk/
    </location>
</VirtualHost>

Removed “permanent”; you had it as 302 temporary

Metricrat · May 11, 2020, 11:54am

OK, tried that, unfortunately the same result, a 404 from the google site.

I am grateful for your help and support

rg305 · May 11, 2020, 11:57am

That is actually expected.
What is NOT expected is the URL with the /.well-known/acme-challenge/ path to also be redirected:

curl -Iki http://ai2.metricrat.co.uk/.well-known/acme-challenge/test-file
HTTP/1.1 301 Moved Permanently
Content-Type: application/binary
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Mon, 11 May 2020 11:55:51 GMT
Location: https://ai2.metricrat.co.uk/.well-known/acme-challenge/test-file
Content-Length: 0
Server: ESF
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff

rg305 · May 11, 2020, 11:58am

Let’s backup a few steps…
[something may have been missed/overlooked]
Please show:
sudo apachectl -S

Metricrat · May 11, 2020, 12:24pm

I have not changed the virtual host settings for ai2.metricrat.co.uk from the last test....

sudo apachectl -S

VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server ai2.metricrat.co.uk (/etc/apache2/sites-enabled/ai2.metricrat.co.uk-le-ssl.conf:2)
         port 443 namevhost ai2.metricrat.co.uk (/etc/apache2/sites-enabled/ai2.metricrat.co.uk-le-ssl.conf:2)
         port 443 namevhost burbush.co.uk (/etc/apache2/sites-enabled/burbush.co.uk-le-ssl.conf:2)
                 alias www.burbush.co.uk
         port 443 namevhost carter-computing.co.uk (/etc/apache2/sites-enabled/carter-computing.co.uk-le-ssl.conf:2)
                 alias www.carter-computing.co.uk
         port 443 namevhost www.cyberama.co.uk (/etc/apache2/sites-enabled/cyberama.co.uk-le-ssl.conf:2)
                 alias cyberama.co.uk
         port 443 namevhost metricrat.co.uk (/etc/apache2/sites-enabled/metricrat.co.uk-le-ssl.conf:2)
                 alias www.metricrat.co.uk
         port 443 namevhost tsah.co.uk (/etc/apache2/sites-enabled/tsah.co.uk-le-ssl.conf:2)
                 alias www.tsah.co.uk
*:80                   is a NameVirtualHost
         default server localhost (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost localhost (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost ai2.metricrat.co.uk (/etc/apache2/sites-enabled/ai2.metricrat.co.uk.conf:1)
         port 80 namevhost burbush.co.uk (/etc/apache2/sites-enabled/burbush.co.uk.conf:1)
                 alias www.burbush.co.uk
         port 80 namevhost carter-computing.co.uk (/etc/apache2/sites-enabled/carter-computing.co.uk.conf:1)
                 alias www.carter-computing.co.uk
         port 80 namevhost www.cyberama.co.uk (/etc/apache2/sites-enabled/cyberama.co.uk.conf:1)
                 alias cyberama.co.uk
         port 80 namevhost metricrat.co.uk (/etc/apache2/sites-enabled/metricrat.co.uk.conf:1)
                 alias www.metricrat.co.uk
         port 80 namevhost tsah.co.uk (/etc/apache2/sites-enabled/tsah.co.uk.conf:1)
                 alias www.tsah.co.uk
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: MODSEC_2.5
Define: MODSEC_2.9
User: name="www-data" id=33
Group: name="www-data" id=33

rg305 · May 11, 2020, 12:28pm

To be sure we are dealing with the correct relevant file…
Please show:
cat /etc/apache2/sites-enabled/ai2.metricrat.co.uk.conf

Metricrat · May 11, 2020, 12:30pm

cat /etc/apache2/sites-enabled/ai2.metricrat.co.uk.conf

<VirtualHost *:80>
    ServerAdmin webmaster@ai2.metricrat.co.uk
    ServerName ai2.metricrat.co.uk
    DocumentRoot /var/www/ai2.metricrat.co.uk/public_html
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
    
    <location "/.well-known/acme-challenge/">
         Alias "/home/metricrat/cert/"
    </location>
    <location "/">
         Redirect / https://ai2.metricrat.co.uk/
    </location>
</VirtualHost>

rg305 · May 11, 2020, 12:36pm

Try it this way:

    <location "/.well-known/acme-challenge/">
         Alias "/home/metricrat/cert/"
         DocumentRoot "/home/metricrat/cert/"
    </location>

or maybe without the last slash and no quotes...
    <location /.well-known/acme-challenge>
         Alias /home/metricrat/cert
    </location>

[I hate Apache]

rg305 · May 11, 2020, 12:43pm

Also please show:
ls -l /home/metricrat/cert
[that directory must exist]

Topic		Replies	Views
Produced an unexpected error: Failed authorization procedure Help	69	13742	January 21, 2019
Unauthorized invalid response from (cannot renew) Help	257	3518	February 26, 2021
Failed authorization procedure Help	23	1348	March 7, 2019
Certificate renewal simulation failure Help	37	496	October 16, 2024
Certificate renewal is failing with missing / Help	132	2803	August 30, 2021

Cannot Renew Certificate - authorisation error

Related topics