Renew domain keeps failing with 404 errors

daviddeleeuw · November 11, 2021, 7:38am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
dukium.org
I ran this command:
certbot renew --dry-run --apache
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/dukium.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dukium.org
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (dukium.org) from /etc/letsencrypt/renewal/dukium.org.conf produced an unexpected error: Failed authorization procedure. dukium.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://dukium.org/.well-known/acme-challenge/i3qzFLTdN0d2BrupRrh_Y7VqaFconV6lrD9WGR66bkI [66.23.235.247]: "\n\n404 Not Found\n\n

Not Found

\n<p". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/dukium.org/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/dukium.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: dukium.org
Type: unauthorized
Detail: Invalid response from
http://dukium.org/.well-known/acme-challenge/i3qzFLTdN0d2BrupRrh_Y7VqaFconV6lrD9WGR66bkI
[66.23.235.247]: "\n\n404 Not
Found\n\n

Not Found
\n<p"

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

My web server is (include version):
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2020-08-12T21:35:50

The operating system my web server runs on is (include version):
Linux dukium.org 4.4.0-1160.41.1.vz7.183.5 #1 SMP Thu Sep 23 18:26:47 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux
My hosting provider, if applicable, is:
interserver.net
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.31.0

The run fails on:
http://dukium.org/.well-known/acme-challenge/i3qzFLTdN0d2BrupRrh_Y7VqaFconV6lrD9WGR66bkI

But
http://dukium.org/.well-known/acme-challenge/a
works !

There is a redirection in place from /var/www to /usr/local/mgr5 for the whole website

There is a "letsencrypt.conf"
Alias /.well-known/acme-challenge/ /usr/local/mgr5/www/letsencrypt/
<Directory "/usr/local/mgr5/www/letsencrypt/">
Order allow,deny
Allow from all
= 2.4>
AllowOverride None
Require all granted

The system ran years ago on another provider with cpanel, was moved. The certificate was provided through cloudflare. The certificate present on the system is from 2017. I was not involved in this this yesterday, so can not really follow the history.
Did not work with certbot before.
Thanks for your help.
David de Leeuw
Israel

rg305 · November 11, 2021, 8:17am

Hi @daviddeleeuw and welcome to the LE community forum

How does that "work"?

Please start by showing the output of:
sudo apachectl -t -D DUMP_VHOSTS

daviddeleeuw · November 11, 2021, 8:19am

access a writes "hello world"

sudo apachectl -t -D DUMP_VHOSTS

VirtualHost configuration:
66.23.235.247:80 is a NameVirtualHost
default server dukium.org (/etc/apache2/vhosts/www-root/dukium.org:1)
port 80 namevhost dukium.org (/etc/apache2/vhosts/www-root/dukium.org:1)
alias www.dukium.org
port 80 namevhost dukium.org (/etc/apache2/vhosts/www-root/dukium.org.orig:1)
alias www.dukium.org
66.23.235.247:443 is a NameVirtualHost
default server dukium.org (/etc/apache2/vhosts/www-root/dukium.org:39)
port 443 namevhost dukium.org (/etc/apache2/vhosts/www-root/dukium.org:39)
alias www.dukium.org
port 443 namevhost dukium.org (/etc/apache2/vhosts/www-root/dukium.org.orig:39)
alias www.dukium.org
*:443 dukium.org (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)
*:80 dukium.org (/etc/apache2/sites-enabled/000-default.conf:1)

Thanks
David

daviddeleeuw · November 11, 2021, 9:30am

I ran
wget --no-check-certificate -S -O/dev/null "http://dukium.org/.well-known/acme-challenge/foo"
--2021-11-11 10:28:30-- http://dukium.org/.well-known/acme-challenge/foo
Resolving dukium.org (dukium.org)... 66.23.235.247
Connecting to dukium.org (dukium.org)|66.23.235.247|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 301 Moved Permanently
Date: Thu, 11 Nov 2021 09:28:30 GMT
Server: Apache/2.4.18 (Ubuntu)
X-Redirect-By: WordPress
Location: https://www.dukium.org/.well-known/acme-challenge/foo
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Location: https://www.dukium.org/.well-known/acme-challenge/foo [following]
--2021-11-11 10:28:31-- https://www.dukium.org/.well-known/acme-challenge/foo
Resolving www.dukium.org (www.dukium.org)... 66.23.235.247
Connecting to www.dukium.org (www.dukium.org)|66.23.235.247|:443... connected.
WARNING: cannot verify www.dukium.org's certificate, issued by 'emailAddress=dukium@shoof.co.il,ST=XX,OU=XX,O=XX,L=XX,CN=dukium.org,C=XX':
Self-signed certificate encountered.
WARNING: certificate common name 'dukium.org' doesn't match requested host name 'www.dukium.org'.
HTTP request sent, awaiting response...
HTTP/1.1 404 Not Found
Date: Thu, 11 Nov 2021 09:28:31 GMT
Server: Apache/2.4.18 (Ubuntu)
Strict-Transport-Security: max-age=31536000; preload
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Link: https://www.dukium.org/wp-json/; rel="https://api.w.org/"
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
2021-11-11 10:28:31 ERROR 404: Not Found.

Two issues obviously (?)
1.redirection to https
2. certificate goes to www.dukium.org instead of dukium.org
Not sure if this points to the issue
David

rg305 · November 11, 2021, 9:38am

There is a name:port overlap/conflict caused because this file is being included into your running config:

daviddeleeuw · November 11, 2021, 10:37am

RG305
Now it is OK ?

apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
66.23.235.247:80 dukium.org (/etc/apache2/vhosts/www-root/dukium.org:1)
66.23.235.247:443 dukium.org (/etc/apache2/vhosts/www-root/dukium.org:39)
*:443 dukium.org (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)
*:80 dukium.org (/etc/apache2/sites-enabled/000-default.conf:1)

Still same problem.
Thanks
David

rg305 · November 11, 2021, 10:42am

No; There still exist name:port conflicts:

daviddeleeuw:

66.23.235.247:80 dukium.org (/etc/apache2/vhosts/www-root/dukium.org:1)
            *:80 dukium.org (/etc/apache2/sites-enabled/000-default.conf:1)

66.23.235.247:443 dukium.org (/etc/apache2/vhosts/www-root/dukium.org:39)
            *:443 dukium.org (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)

daviddeleeuw · November 11, 2021, 11:14am

hi rg305

apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
66.23.235.247:80 dukium.org (/etc/apache2/vhosts/www-root/dukium.org:1)
66.23.235.247:443 dukium.org (/etc/apache2/vhosts/www-root/dukium.org:39)

now i get

certbot renew --dry-run --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/dukium.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dukium.org
Cleaning up challenges
Attempting to renew cert (dukium.org) from /etc/letsencrypt/renewal/dukium.org.conf produced an unexpected error: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/dukium.org/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/dukium.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)
Going forward or backward ?
Thanks
David

daviddeleeuw · November 11, 2021, 11:17am

There is a redirection from http://dukium.org to https://www.dukium.org . Performed by Wordpress. Should probably stop this redirect for letsencrypt. Where ?

rg305 · November 11, 2021, 11:18am

Definitely forward

Edit that file and change both lines:

To:
*:80
*:443

rg305 · November 11, 2021, 11:19am

Thats only a problem if the two vhosts have different DocumentRoots.

daviddeleeuw · November 11, 2021, 11:22am

apachectl -t -D DUMP_VHOSTS

VirtualHost configuration:
*:80                   dukium.org (/etc/apache2/vhosts/www-root/dukium.org:1)
*:443                  dukium.org (/etc/apache2/vhosts/www-root/dukium.org:39)

certbot renew --dry-run --apache

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/dukium.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dukium.org
Cleaning up challenges
Attempting to renew cert (dukium.org) from /etc/letsencrypt/renewal/dukium.org.conf produced an unexpected error: Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/dukium.org/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/dukium.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

rg305 · November 11, 2021, 11:25am

hmm...
Let's have a look at the file:

daviddeleeuw · November 11, 2021, 11:28am

<VirtualHost *:80>
        ServerName dukium.org
        DocumentRoot /var/www/www-root/data/www/dukium.org
        ServerAdmin office@dukium.org
        AddDefaultCharset UTF-8
        AssignUserID www-root www-root
        CustomLog /var/www/httpd-logs/dukium.org.access.log combined
        ErrorLog /var/www/httpd-logs/dukium.org.error.log
        <FilesMatch "\.ph(p[3-5]?|tml)$">
                SetHandler application/x-httpd-php
        </FilesMatch>
        <FilesMatch "\.phps$">
                SetHandler application/x-httpd-php-source
        </FilesMatch>
        <IfModule php5_module>
                php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f dukium@shoof.co.il"
                php_admin_value upload_tmp_dir "/var/www/www-root/data/mod-tmp"
                php_admin_value session.save_path "/var/www/www-root/data/mod-tmp"
                php_admin_value open_basedir "/var/www/www-root/data:."
        </IfModule>
        <IfModule php7_module>
                php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f dukium@shoof.co.il"
                php_admin_value upload_tmp_dir "/var/www/www-root/data/mod-tmp"
                php_admin_value session.save_path "/var/www/www-root/data/mod-tmp"
                php_admin_value open_basedir "/var/www/www-root/data:."
        </IfModule>
        ServerAlias www.dukium.org
        DirectoryIndex index.php index.html
</VirtualHost>
<Directory /var/www/www-root/data/www/dukium.org>
        Options +Includes -ExecCGI
        <IfModule php5_module>
                php_admin_flag engine on
        </IfModule>
        <IfModule php7_module>
                php_admin_flag engine on
        </IfModule>
</Directory>
 and the same for 
<VirtualHost *:443>

daviddeleeuw · November 11, 2021, 11:28am

I notice there is no redirect exclusion for letsencrypt here. Is that the problem ?

rg305 · November 11, 2021, 11:30am

Let me check that.
Yeah, the WordPress is redirecting to HTTPS.
Please show the 443 section.

daviddeleeuw · November 11, 2021, 11:43am

<VirtualHost *:443>
        ServerName dukium.org
        DocumentRoot /var/www/www-root/data/www/dukium.org
        ServerAdmin office@dukium.org
        AddDefaultCharset UTF-8
        SSLEngine on
        SSLCertificateFile "/var/www/httpd-cert/www-root/dukium.org_le2.crt"
        SSLCertificateKeyFile "/var/www/httpd-cert/www-root/dukium.org_le2.key"
        SSLHonorCipherOrder on
        SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
        SSLCipherSuite EECDH:+AES256:-3DES:RSA+AES:RSA+3DES:!NULL:!RC4:!RSA+3DES
        <IfModule headers_module>
                Header always set Strict-Transport-Security "max-age=31536000; preload"
        </IfModule>
        AssignUserID www-root www-root
        CustomLog /var/www/httpd-logs/dukium.org.access.log combined
        ErrorLog /var/www/httpd-logs/dukium.org.error.log
        <FilesMatch "\.ph(p[3-5]?|tml)$">
                SetHandler application/x-httpd-php
        </FilesMatch>
        <FilesMatch "\.phps$">
                SetHandler application/x-httpd-php-source
        </FilesMatch>
        <IfModule php5_module>
                php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f dukium@shoof.co.il"
                php_admin_value upload_tmp_dir "/var/www/www-root/data/mod-tmp"
                php_admin_value session.save_path "/var/www/www-root/data/mod-tmp"
                php_admin_value open_basedir "/var/www/www-root/data:."
        </IfModule>
        <IfModule php7_module>
                php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f dukium@shoof.co.il"
                php_admin_value upload_tmp_dir "/var/www/www-root/data/mod-tmp"
                php_admin_value session.save_path "/var/www/www-root/data/mod-tmp"
                php_admin_value open_basedir "/var/www/www-root/data:."
        </IfModule>
        ServerAlias www.dukium.org
        DirectoryIndex index.php index.html
</VirtualHost>

daviddeleeuw · November 11, 2021, 11:44am

Should these ssl lines be there ? Seem to point to other places ?

rg305 · November 11, 2021, 11:47am

Ok. Let's try it this way:

certbot --dry-run --apache \
--webroot -w /var/www/www-root/data/www/dukium.org \
-d "dukium.org,www.dukium.org"

Which may fail with the same error...
If so, then try:

certbot --dry-run certonly \
--webroot -w /var/www/www-root/data/www/dukium.org \
-d "dukium.org,www.dukium.org"

daviddeleeuw · November 11, 2021, 11:51am

certbot renew --dry-run --apache --webroot -w /var/www/www-root/data/www/dukium.org -d "dukium.org,www.dukium.org"

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future.

certbot certonly --dry-run --apache --webroot -w /var/www/www-root/data/www/dukium.org -d "dukium.org,www.dukium.org"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Could not choose appropriate plugin: Too many flags setting configurators/installers/authenticators 'apache' -> 'webroot'
Too many flags setting configurators/installers/authenticators 'apache' -> 'webroot'