Unable to renew certificates


#1

Working domain: chat.icecub.nl
Domains not working: icecub.nl / www.icecub.nl / xgn-gaming.com / www.xgn-gaming.com

I ran this command: certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/icecub.nl.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for icecub.nl
http-01 challenge for www.icecub.nl
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (icecub.nl) from /etc/letsencrypt/renewal/icecub.nl.conf produced an unexpected error: Failed authorization procedure. icecub.nl (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://icecub.nl/.well-known/acme-challenge/uouOpV2KmQ-saf4vJynG9Q6NlP0Npn-GmXD4rGX5x7w: "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<ht", www.icecub.nl (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.icecub.nl/.well-known/acme-challenge/SREMWMMKm5lDHHe9vCcyiIcCRN7lDMbQKW0zRN0FFbc: "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<ht". Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/chat.icecub.nl.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for chat.icecub.nl
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/chat.icecub.nl/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/xgn-gaming.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for xgn-gaming.com
http-01 challenge for www.xgn-gaming.com
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/xgn-gaming.com/web/.well-known/acme-challenge
Attempting to renew cert (xgn-gaming.com) from /etc/letsencrypt/renewal/xgn-gaming.com.conf produced an unexpected error: Failed authorization procedure. xgn-gaming.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://xgn-gaming.com/.well-known/acme-challenge/8zxfrHzQmcUhbXdiuAOf21YZMRNdZbnr3d-k3i94s08: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p", www.xgn-gaming.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.xgn-gaming.com/.well-known/acme-challenge/8VJWSCXi-4hct1e3QBMmLYW9Pjibtep8CTuc9cGrQ88: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p". Skipping.
The following certs could not be renewed:
  /etc/letsencrypt/live/icecub.nl/fullchain.pem (failure)
  /etc/letsencrypt/live/xgn-gaming.com/fullchain.pem (failure)

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

The following certs were successfully renewed:
  /etc/letsencrypt/live/chat.icecub.nl/fullchain.pem (success)

The following certs could not be renewed:
  /etc/letsencrypt/live/icecub.nl/fullchain.pem (failure)
  /etc/letsencrypt/live/xgn-gaming.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: icecub.nl
   Type:   unauthorized
   Detail: Invalid response from
   http://icecub.nl/.well-known/acme-challenge/uouOpV2KmQ-saf4vJynG9Q6NlP0Npn-GmXD4rGX5x7w:
   "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   <ht"

   Domain: www.icecub.nl
   Type:   unauthorized
   Detail: Invalid response from
   http://www.icecub.nl/.well-known/acme-challenge/SREMWMMKm5lDHHe9vCcyiIcCRN7lDMbQKW0zRN0FFbc:
   "<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   <ht"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: xgn-gaming.com
   Type:   unauthorized
   Detail: Invalid response from
   http://xgn-gaming.com/.well-known/acme-challenge/8zxfrHzQmcUhbXdiuAOf21YZMRNdZbnr3d-k3i94s08:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   Domain: www.xgn-gaming.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.xgn-gaming.com/.well-known/acme-challenge/8VJWSCXi-4hct1e3QBMmLYW9Pjibtep8CTuc9cGrQ88:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): Apache/2.4.18

The operating system my web server runs on is (include version): Ubuntu 16.04.3 LTS (Xenial Xerus)

My hosting provider, if applicable, is: Self hosted

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): ISPConfig 3 is installed.


#2

First, try using certbot with “sudo”:
sudo certbot renew --dry-run

That may overcome the “Unable to clean up challenge directory…” problem.


#3

It looks like the webroot directory that you chose when you originally obtained these certificates (you can find it in /etc/letsencrypt/renewal/icecub.nl.conf and /etc/letsencrypt/renewal/xgn-gaming.com.conf) no longer works as a place to put static files and have them appear on your web site. Perhaps you’ve changed your web site configuration since obtaining the certificates by installing a CMS or something?

Can you take a look at the webroot directory configurations and see if they’re still correct and if files placed there still appear on the web site?


#4

All commands were performed after logging in as root with sudo -s


#5

I’ve checked the files you advised, they’re showing me:

[[webroot_map]]
www.icecub.nl = /var/www/icecub.nl/web
icecub.nl = /var/www/icecub.nl/web

After varifying my apache config files, they’re showing:

            <Directory /var/www/icecub.nl/web>
                            # Clear PHP settings of this website
                            <FilesMatch ".+\.ph(p[345]?|t|tml)$">
                                            SetHandler None
                            </FilesMatch>
                            Options +FollowSymLinks
                            AllowOverride All
                                                            Require all granted

                            # ssi enabled
                            AddType text/html .shtml
                            AddOutputFilter INCLUDES .shtml
                            Options +Includes
            </Directory>

What might be an issue, though not sure as it worked perfectly fine while I got the initial certificates, is that that’s a symlink towards the actual webroot, which is:

            <Directory /var/www/clients/client1/web4/web>
                            # Clear PHP settings of this website
                            <FilesMatch ".+\.ph(p[345]?|t|tml)$">
                                            SetHandler None
                            </FilesMatch>
                            Options +FollowSymLinks
                            AllowOverride All
                                                            Require all granted

                            # ssi enabled
                            AddType text/html .shtml
                            AddOutputFilter INCLUDES .shtml
                            Options +Includes
            </Directory>

But that goes for chat.icecub.nl just as well and that one is working fine.


#6

What happens if you put a file in /var/www/icecub.nl/web/.well-known/acme-challenge/test.txt?


#7

The console verifies the file is there:

root@server:/var/www/icecub.nl/web/.well-known/acme-challenge# ls
test.txt
root@server:/var/www/icecub.nl/web/.well-known/acme-challenge#

I made sure to chmod the dir recursively to 0755 and open it up in the browser, but a 404 error is returned: https://icecub.nl/.well-known/acme-challenge/test.txt


#8

Where are you doing the http to https redirection?
show that block.

and/or search your error logs for access to the test.txt file


#9

I’m also curious about this. In addition, do you have a separate HTTPS virtual host that might have a different web root from the HTTP virtual host?


#10

There may be some unexpected overlap…
grep -Eri 'servername|serveralias' /etc/apache2


#11

The redirect to https is done here:

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.icecub.nl [OR]
RewriteCond %{SERVER_NAME} =icecub.nl
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

This was added by Certbot itself. I had no idea how to do this so I simply let Certbot handle it.


#12

That command returned:

root@server:/etc/apache2/sites-enabled# grep -Eri 'servername|serveralias' /etc/apache2
/etc/apache2/mods-available/info.conf:  #  http://servername/server-info (requires that mod_info.c be loaded).
/etc/apache2/mods-available/status.conf:        # with the URL of http://servername/server-status
/etc/apache2/sites-available/000-default.conf:  # The ServerName directive sets the request scheme, hostname and port that
/etc/apache2/sites-available/000-default.conf:  # redirection URLs. In the context of virtual hosts, the ServerName
/etc/apache2/sites-available/000-default.conf:  #ServerName www.example.com
/etc/apache2/sites-available/home.icecub.nl.vhost:              ServerName home.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerName chat.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerAlias www.chat.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerName chat.icecub.nl
/etc/apache2/sites-available/chat.icecub.nl.vhost:              ServerAlias www.chat.icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerName icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerAlias www.icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerName icecub.nl
/etc/apache2/sites-available/icecub.nl.vhost:           ServerAlias www.icecub.nl
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerName xgn-gaming.com
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerAlias www.xgn-gaming.com
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerName xgn-gaming.com
/etc/apache2/sites-available/xgn-gaming.com.vhost:              ServerAlias www.xgn-gaming.com

#13

Please show this file:
/etc/apache2/sites-available/icecub.nl.vhost
(the FQDNs appear twice)

Also:

ls -l /etc/apache2/sites-enabled/


#14

As the file is quite large, I made a pastebin of it here: https://pastebin.com/LFXazyfe It will expire in a day.

The result of that command was:

root@server:/etc/apache2/sites-available# ls -l /etc/apache2/sites-enabled/
total 0
lrwxrwxrwx 1 root root 39 Dec 12 11:36 000-apps.vhost -> /etc/apache2/sites-available/apps.vhost
lrwxrwxrwx 1 root root 35 Dec 12 11:00 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root root 43 Dec 12 11:35 000-ispconfig.conf -> /etc/apache2/sites-available/ispconfig.conf
lrwxrwxrwx 1 root root 44 Dec 12 11:37 000-ispconfig.vhost -> /etc/apache2/sites-available/ispconfig.vhost
lrwxrwxrwx 1 root root 49 Jan 17 00:27 100-chat.icecub.nl.vhost -> /etc/apache2/sites-available/chat.icecub.nl.vhost
lrwxrwxrwx 1 root root 49 Dec 12 11:46 100-home.icecub.nl.vhost -> /etc/apache2/sites-available/home.icecub.nl.vhost
lrwxrwxrwx 1 root root 44 Jan 22 17:45 100-icecub.nl.vhost -> /etc/apache2/sites-available/icecub.nl.vhost
lrwxrwxrwx 1 root root 49 Dec 14 17:04 100-xgn-gaming.com.vhost -> /etc/apache2/sites-available/xgn-gaming.com.vhost

#15

FYI: Those are processed in alphabetical order.
So please show those that are listed above icecub.nl.vhost

I’m thinking the problem is within the first four:
000-apps.vhost
000-default.conf
000-ispconfig.conf
000-ispconfig.vhost


#16

Doesn’t match up with:
DocumentRoot /var/www/clients/client1/web4/web

Please show:
/etc/letsencrypt/renewal/icecub.nl.conf

And try placing a test.txt file at:
/var/www/clients/client1/web4/web/.well-known/acme-challenge/


#17

000-apps.vhost: https://pastebin.com/kLACMUWQ
000-default.conf: https://pastebin.com/iDqz9rut
000-ispconfig.conf: https://pastebin.com/TUh0uupt
000-ispconfig.vhost: https://pastebin.com/wLyjitQT

As I’ve said earlier, /var/www/icecub.nl/ is a symlink to /var/www/clients/client1/web4/


#18

Nothing funny looking in those four files.
hmm…
[edit] yes, sorry, I missed the earlier sym link reference

Check the apache error logs for the test.txt failed attempts.

And let’s look for any potential acme challenge handling:
grep -Eri 'well-known|challenge' /etc/apache2


#19

Please also show:
/etc/letsencrypt/renewal/icecub.nl.conf


#20

The error log doesn’t tell me much. All it returns is:

[Wed May 23 04:27:39.238713 2018] [ssl:error] [pid 24827] AH01936: stapling_check_response: response times invalid
[Wed May 23 04:27:39.238767 2018] [ssl:error] [pid 24827] AH01943: stapling_renew_response: error in retrieved response!

The command returned:

root@server:/var/log/apache2# grep -Eri 'well-known|challenge' /etc/apache2
/etc/apache2/sites-available/ispconfig.conf:Alias /.well-known/acme-challenge /usr/local/ispconfig/interface/acme/.well-known/acme-challenge
/etc/apache2/sites-available/ispconfig.conf:<Directory /usr/local/ispconfig/interface/acme/.well-known/acme-challenge>
/etc/apache2/sites-available/home.icecub.nl.vhost:                Alias /.well-known/acme-challenge/ /var/www/home.icecub.nl/web/.well-known/acme-challenge/
/etc/apache2/sites-available/home.icecub.nl.vhost:                <Directory /var/www/home.icecub.nl/web/.well-known/acme-challenge>
/etc/apache2/sites-available/home.icecub.nl.vhost:                                RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
/etc/apache2/sites-available/chat.icecub.nl.vhost:                Alias /.well-known/acme-challenge/ /var/www/chat.icecub.nl/web/.well-known/acme-challenge/
/etc/apache2/sites-available/chat.icecub.nl.vhost:                <Directory /var/www/chat.icecub.nl/web/.well-known/acme-challenge>
/etc/apache2/sites-available/chat.icecub.nl.vhost:                                RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
/etc/apache2/sites-available/icecub.nl.vhost:                Alias /.well-known/acme-challenge/ /var/www/icecub.nl/web/.well-known/acme-challenge/
/etc/apache2/sites-available/icecub.nl.vhost:                <Directory /var/www/icecub.nl/web/.well-known/acme-challenge>
/etc/apache2/sites-available/icecub.nl.vhost:                                RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"
/etc/apache2/sites-available/xgn-gaming.com.vhost:              Alias /.well-known/acme-challenge/ /var/www/xgn-gaming.com/web/.well-known/acme-challenge/
/etc/apache2/sites-available/xgn-gaming.com.vhost:              <Directory /var/www/xgn-gaming.com/web/.well-known/acme-challenge>
/etc/apache2/sites-available/xgn-gaming.com.vhost:                               RedirectMatch 404 "^(?!/\.well-known/acme-challenge/[\w-]{43}$)"

Those are part of a fix I applied 3 months ago to get the certificates working. I’ve tried removing those blocks earlier but the same issue arrises.

The icecub.nl.conf file contains:

# renew_before_expiry = 30 days
cert = /etc/letsencrypt/live/icecub.nl/cert.pem
privkey = /etc/letsencrypt/live/icecub.nl/privkey.pem
chain = /etc/letsencrypt/live/icecub.nl/chain.pem
fullchain = /etc/letsencrypt/live/icecub.nl/fullchain.pem
version = 0.19.0
archive_dir = /etc/letsencrypt/archive/icecub.nl

# Options and defaults used in the renewal process
[renewalparams]
installer = apache
authenticator = webroot
account = 7c36e6c37c0756f652a3dcca69903bf9
[[webroot_map]]
www.icecub.nl = /var/www/icecub.nl/web
icecub.nl = /var/www/icecub.nl/web