Unable to renew certificates

Working domain: chat.icecub.nl
Domains not working: icecub.nl / www.icecub.nl / xgn-gaming.com / www.xgn-gaming.com

I ran this command: certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/icecub.nl.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for icecub.nl
http-01 challenge for www.icecub.nl
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (icecub.nl) from /etc/letsencrypt/renewal/icecub.nl.conf produced an unexpected error: Failed authorization procedure. icecub.nl (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://icecub.nl/.well-known/acme-challenge/uouOpV2KmQ-saf4vJynG9Q6NlP0Npn-GmXD4rGX5x7w: "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<ht", www.icecub.nl (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.icecub.nl/.well-known/acme-challenge/SREMWMMKm5lDHHe9vCcyiIcCRN7lDMbQKW0zRN0FFbc: "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<ht". Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/chat.icecub.nl.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for chat.icecub.nl
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/chat.icecub.nl/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/xgn-gaming.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for xgn-gaming.com
http-01 challenge for www.xgn-gaming.com
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/xgn-gaming.com/web/.well-known/acme-challenge
Attempting to renew cert (xgn-gaming.com) from /etc/letsencrypt/renewal/xgn-gaming.com.conf produced an unexpected error: Failed authorization procedure. xgn-gaming.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://xgn-gaming.com/.well-known/acme-challenge/8zxfrHzQmcUhbXdiuAOf21YZMRNdZbnr3d-k3i94s08: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p", www.xgn-gaming.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.xgn-gaming.com/.well-known/acme-challenge/8VJWSCXi-4hct1e3QBMmLYW9Pjibtep8CTuc9cGrQ88: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p". Skipping.
The following certs could not be renewed:
  /etc/letsencrypt/live/icecub.nl/fullchain.pem (failure)
  /etc/letsencrypt/live/xgn-gaming.com/fullchain.pem (failure)

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

The following certs were successfully renewed:
  /etc/letsencrypt/live/chat.icecub.nl/fullchain.pem (success)

The following certs could not be renewed:
  /etc/letsencrypt/live/icecub.nl/fullchain.pem (failure)
  /etc/letsencrypt/live/xgn-gaming.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: icecub.nl
   Type:   unauthorized
   Detail: Invalid response from
   http://icecub.nl/.well-known/acme-challenge/uouOpV2KmQ-saf4vJynG9Q6NlP0Npn-GmXD4rGX5x7w:
   "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   <ht"

   Domain: www.icecub.nl
   Type:   unauthorized
   Detail: Invalid response from
   http://www.icecub.nl/.well-known/acme-challenge/SREMWMMKm5lDHHe9vCcyiIcCRN7lDMbQKW0zRN0FFbc:
   "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   <ht"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: xgn-gaming.com
   Type:   unauthorized
   Detail: Invalid response from
   http://xgn-gaming.com/.well-known/acme-challenge/8zxfrHzQmcUhbXdiuAOf21YZMRNdZbnr3d-k3i94s08:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   Domain: www.xgn-gaming.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.xgn-gaming.com/.well-known/acme-challenge/8VJWSCXi-4hct1e3QBMmLYW9Pjibtep8CTuc9cGrQ88:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): Apache/2.4.18

The operating system my web server runs on is (include version): Ubuntu 16.04.3 LTS (Xenial Xerus)

My hosting provider, if applicable, is: Self hosted

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): ISPConfig 3 is installed.

First, try using certbot with "sudo":
sudo certbot renew --dry-run

That may overcome the "Unable to clean up challenge directory..." problem.

It looks like the webroot directory that you chose when you originally obtained these certificates (you can find it in /etc/letsencrypt/renewal/icecub.nl.conf and /etc/letsencrypt/renewal/xgn-gaming.com.conf) no longer works as a place to put static files and have them appear on your web site. Perhaps you’ve changed your web site configuration since obtaining the certificates by installing a CMS or something?

Can you take a look at the webroot directory configurations and see if they’re still correct and if files placed there still appear on the web site?

All commands were performed after logging in as root with sudo -s

I’ve checked the files you advised, they’re showing me:

[[webroot_map]]
www.icecub.nl = /var/www/icecub.nl/web
icecub.nl = /var/www/icecub.nl/web

After varifying my apache config files, they’re showing:

            <Directory /var/www/icecub.nl/web>
                            # Clear PHP settings of this website
                            <FilesMatch ".+\.ph(p[345]?|t|tml)$">
                                            SetHandler None
                            </FilesMatch>
                            Options +FollowSymLinks
                            AllowOverride All
                                                            Require all granted

                            # ssi enabled
                            AddType text/html .shtml
                            AddOutputFilter INCLUDES .shtml
                            Options +Includes
            </Directory>

What might be an issue, though not sure as it worked perfectly fine while I got the initial certificates, is that that’s a symlink towards the actual webroot, which is:

            <Directory /var/www/clients/client1/web4/web>
                            # Clear PHP settings of this website
                            <FilesMatch ".+\.ph(p[345]?|t|tml)$">
                                            SetHandler None
                            </FilesMatch>
                            Options +FollowSymLinks
                            AllowOverride All
                                                            Require all granted

                            # ssi enabled
                            AddType text/html .shtml
                            AddOutputFilter INCLUDES .shtml
                            Options +Includes
            </Directory>

But that goes for chat.icecub.nl just as well and that one is working fine.

What happens if you put a file in /var/www/icecub.nl/web/.well-known/acme-challenge/test.txt?

The console verifies the file is there:

root@server:/var/www/icecub.nl/web/.well-known/acme-challenge# ls
test.txt
root@server:/var/www/icecub.nl/web/.well-known/acme-challenge#

I made sure to chmod the dir recursively to 0755 and open it up in the browser, but a 404 error is returned: https://icecub.nl/.well-known/acme-challenge/test.txt

Where are you doing the http to https redirection?
show that block.

and/or search your error logs for access to the test.txt file

I'm also curious about this. In addition, do you have a separate HTTPS virtual host that might have a different web root from the HTTP virtual host?

1 Like

There may be some unexpected overlap…
grep -Eri 'servername|serveralias' /etc/apache2

The redirect to https is done here:

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.icecub.nl [OR]
RewriteCond %{SERVER_NAME} =icecub.nl
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

This was added by Certbot itself. I had no idea how to do this so I simply let Certbot handle it.

That command returned:

root@server:/etc/apache2/sites-enabled# grep -Eri 'servername|serveralias' /etc/apache2
/etc/apache2/mods-available/info.conf:  #  http://servername/server-info (requires that mod_info.c be loaded).
/etc/apache2/mods-available/status.conf:        # with the URL of http://servername/server-status
/etc/apache2/sites-available/000-default.conf:  # The ServerName directive sets the request scheme, hostname and port that
/etc/apache2/sites-available/000-default.conf:  # redirection URLs. In the context of virtual hosts, the ServerName
/etc/apache2/sites-available/000-default.conf:  #ServerName www.example.com
/etc/apache2/sites-available/home.icecub.nl.vhost:              ServerName home.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerName chat.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerAlias www.chat.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerName chat.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerAlias www.chat.icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerName icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerAlias www.icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerName icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerAlias www.icecub.nl
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerName xgn-gaming.com
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerAlias www.xgn-gaming.com
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerName xgn-gaming.com
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerAlias www.xgn-gaming.com

Please show this file:
/etc/apache2/sites-available/icecub.nl.vhost
(the FQDNs appear twice)

Also:

ls -l /etc/apache2/sites-enabled/

As the file is quite large, I made a pastebin of it here: https://pastebin.com/LFXazyfe It will expire in a day.

The result of that command was:

root@server:/etc/apache2/sites-available# ls -l /etc/apache2/sites-enabled/
total 0
lrwxrwxrwx 1 root root 39 Dec 12 11:36 000-apps.vhost -> /etc/apache2/sites-available/apps.vhost
lrwxrwxrwx 1 root root 35 Dec 12 11:00 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root root 43 Dec 12 11:35 000-ispconfig.conf -> /etc/apache2/sites-available/ispconfig.conf
lrwxrwxrwx 1 root root 44 Dec 12 11:37 000-ispconfig.vhost -> /etc/apache2/sites-available/ispconfig.vhost
lrwxrwxrwx 1 root root 49 Jan 17 00:27 100-chat.icecub.nl.vhost -> /etc/apache2/sites-available/chat.icecub.nl.vhost
lrwxrwxrwx 1 root root 49 Dec 12 11:46 100-home.icecub.nl.vhost -> /etc/apache2/sites-available/home.icecub.nl.vhost
lrwxrwxrwx 1 root root 44 Jan 22 17:45 100-icecub.nl.vhost -> /etc/apache2/sites-available/icecub.nl.vhost
lrwxrwxrwx 1 root root 49 Dec 14 17:04 100-xgn-gaming.com.vhost -> /etc/apache2/sites-available/xgn-gaming.com.vhost

FYI: Those are processed in alphabetical order.
So please show those that are listed above icecub.nl.vhost

I’m thinking the problem is within the first four:
000-apps.vhost
000-default.conf
000-ispconfig.conf
000-ispconfig.vhost

Doesn't match up with:
DocumentRoot /var/www/clients/client1/web4/web

Please show:
/etc/letsencrypt/renewal/icecub.nl.conf

And try placing a test.txt file at:
/var/www/clients/client1/web4/web/.well-known/acme-challenge/

000-apps.vhost: https://pastebin.com/kLACMUWQ
000-default.conf: https://pastebin.com/iDqz9rut
000-ispconfig.conf: https://pastebin.com/TUh0uupt
000-ispconfig.vhost: https://pastebin.com/wLyjitQT

As I’ve said earlier, /var/www/icecub.nl/ is a symlink to /var/www/clients/client1/web4/

Nothing funny looking in those four files.
hmm…
[edit] yes, sorry, I missed the earlier sym link reference

Check the apache error logs for the test.txt failed attempts.

And let’s look for any potential acme challenge handling:
grep -Eri 'well-known|challenge' /etc/apache2

Please also show:
/etc/letsencrypt/renewal/icecub.nl.conf

The error log doesn’t tell me much. All it returns is:

[Wed May 23 04:27:39.238713 2018] [ssl:error] [pid 24827] AH01936: stapling_check_response: response times invalid
[Wed May 23 04:27:39.238767 2018] [ssl:error] [pid 24827] AH01943: stapling_renew_response: error in retrieved response!

The command returned:

root@server:/var/log/apache2# grep -Eri 'well-known|challenge' /etc/apache2
/etc/apache2/sites-available/ispconfig.conf:Alias /.well-known/acme-challenge /usr/local/ispconfig/interface/acme/.well-known/acme-challenge
/etc/apache2/sites-available/ispconfig.conf:<Directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge>
/etc/apache2/sites-available/home.icecub.nl.vhost:                Alias /.well-known/acme-challenge/ /var/www/home.icecub.nl/web/.well-known/acme-challenge/
/etc/apache2/sites-available/home.icecub.nl.vhost:                <Directory /var/www/home.icecub.nl/web/.well-known/acme-challenge>
/etc/apache2/sites-available/home.icecub.nl.vhost:                                RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
/etc/apache2/sites-available/chat.icecub.nl.vhost:                Alias /.well-known/acme-challenge/ /var/www/chat.icecub.nl/web/.well-known/acme-challenge/
/etc/apache2/sites-available/chat.icecub.nl.vhost:                <Directory /var/www/chat.icecub.nl/web/.well-known/acme-challenge>
/etc/apache2/sites-available/chat.icecub.nl.vhost:                                RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
/etc/apache2/sites-available/icecub.nl.vhost:                Alias /.well-known/acme-challenge/ /var/www/icecub.nl/web/.well-known/acme-challenge/
/etc/apache2/sites-available/icecub.nl.vhost:                <Directory /var/www/icecub.nl/web/.well-known/acme-challenge>
/etc/apache2/sites-available/icecub.nl.vhost:                                RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
/etc/apache2/sites-available/xgn-gaming.com.vhost:              Alias /.well-known/acme-challenge/ /var/www/xgn-gaming.com/web/.well-known/acme-challenge/
/etc/apache2/sites-available/xgn-gaming.com.vhost:              <Directory /var/www/xgn-gaming.com/web/.well-known/acme-challenge>
/etc/apache2/sites-available/xgn-gaming.com.vhost:                               RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"

Those are part of a fix I applied 3 months ago to get the certificates working. I’ve tried removing those blocks earlier but the same issue arrises.

The icecub.nl.conf file contains:

# renew_before_expiry = 30 days
cert = /etc/letsencrypt/live/icecub.nl/cert.pem
privkey = /etc/letsencrypt/live/icecub.nl/privkey.pem
chain = /etc/letsencrypt/live/icecub.nl/chain.pem
fullchain = /etc/letsencrypt/live/icecub.nl/fullchain.pem
version = 0.19.0
archive_dir = /etc/letsencrypt/archive/icecub.nl

# Options and defaults used in the renewal process
[renewalparams]
installer = apache
authenticator = webroot
account = 7c36e6c37c0756f652a3dcca69903bf9
[[webroot_map]]
www.icecub.nl = /var/www/icecub.nl/web
icecub.nl = /var/www/icecub.nl/web