Recurring Invalid response / The client lacks sufficient authorization

My domain is: zoneblue.org
I ran this command: certbot renew

It produced this output: errors relating to Invalid response / The client lacks sufficient authorization

My web server is (include version): apache 2.4.25
The OS: (debian stretch)
My hosting provider: n/a
I can login to a root shell : yes
I'm using a control panel: no
The version of my client is: certbot 0.28.0 (the deb version)

THE BASIC ISSUE

I have had trouble with this one cert before (all the others are fine), and i resolved it by a2dissite the ssl version of the site, deleting the cert , then getting a new one. So having just gone round this loop again, i would love to solve it properly. According to some sources this can be caused by htaccess redirects (eg from www.domain to domain), so last time i added a rule at the top of the htaccess

RewriteRule ^\.well-known\/acme-challenge\/ - [L]

But it didnt seem to help

The certificate is this one

 Certificate Name: zoneblue.org
    Domains: zoneblue.org map.zoneblue.org vps.zoneblue.org webmail.zoneblue.org www.zoneblue.nz www.zoneblue.org zoneblue.nz
    Expiry Date: 2022-04-13 19:42:56+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/zoneblue.org/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/zoneblue.org/privkey.pem

These domains point (using DNS and htaccess) as follows:

zoneblue.org     --> www.zoneblue.nz
zoneblue.nz      --> www.zoneblue.nz
www.zoneblue.org --> www.zoneblue.nz
www.zoneblue.nz  --> www.zoneblue.nz
webmail.zoneblue.org -> webmail.zoneblue.org
map.zoneblue.org     -> map.zoneblue.org
vps.zoneblue.org     -> vps.zoneblue.org #but only used by postfix

From previous experience i need to include the aliases in the cert to avoid users seeing "security risk" issues. Should i split this into 4 certs?

Id appreciate any ideas here, TIA.

2 Likes

Welcome to the Let's Encrypt Community, Peter :slightly_smiling_face:

In general, you should avoid htaccess redirects like the plague. It is far more efficient to use apache configuration redirects.

What are the contents of your current htaccess files?

What is the output of:

sudo apachectl -S
3 Likes
vps:/etc/letsencrypt/# apachectl -S

VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server kinesiology.lifeiswonderful.nz (/etc/apache2/sites-enabled/kinesiology.lifeiswonderful.nz-le-ssl.conf:2)
         port 443 namevhost kinesiology.lifeiswonderful.nz (/etc/apache2/sites-enabled/kinesiology.lifeiswonderful.nz-le-ssl.conf:2)
         port 443 namevhost lifeiswonderful.nz (/etc/apache2/sites-enabled/lifeiswonderful.nz-le-ssl.conf:2)
                 alias www.lifeiswonderful.nz
         port 443 namevhost map.zoneblue.org (/etc/apache2/sites-enabled/map.zoneblue.org-le-ssl.conf:2)
         port 443 namevhost vps.zoneblue.org (/etc/apache2/sites-enabled/vps.zoneblue.org-le-ssl.conf:2)
         port 443 namevhost webmail.zoneblue.org (/etc/apache2/sites-enabled/webmail.zoneblue.org-le-ssl.conf:2)
         port 443 namevhost www.webspaces.net.nz (/etc/apache2/sites-enabled/www.webspaces.net.nz-le-ssl.conf:3)
                 alias webspaces.net.nz
         port 443 namevhost www.zoneblue.org (/etc/apache2/sites-enabled/www.zoneblue.org-le-ssl.conf:3)
                 alias zoneblue.org
                 alias zoneblue.nz
                 alias www.zoneblue.nz
*:80                   is a NameVirtualHost
         default server www.example.org (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost www.example.org (/etc/apache2/sites-enabled/000-default.conf:1)
                 alias example.org
         port 80 namevhost kinesiology.lifeiswonderful.nz (/etc/apache2/sites-enabled/kinesiology.lifeiswonderful.nz.conf:1)
         port 80 namevhost lifeiswonderful.nz (/etc/apache2/sites-enabled/lifeiswonderful.nz.conf:1)
                 alias www.lifeiswonderful.nz
         port 80 namevhost map.zoneblue.org (/etc/apache2/sites-enabled/map.zoneblue.org-le-ssl.conf:23)
         port 80 namevhost map.zoneblue.org (/etc/apache2/sites-enabled/map.zoneblue.org-le-ssl.conf:42)
         port 80 namevhost map.zoneblue.org (/etc/apache2/sites-enabled/map.zoneblue.org-le-ssl.conf:61)
         port 80 namevhost map.zoneblue.org (/etc/apache2/sites-enabled/map.zoneblue.org-le-ssl.conf:80)
         port 80 namevhost map.zoneblue.org (/etc/apache2/sites-enabled/map.zoneblue.org.conf:1)
         port 80 namevhost thebighouseproject.nz (/etc/apache2/sites-enabled/thebighouseproject.nz.conf:1)
                 alias www.thebighouseproject.nz
         port 80 namevhost vps.zoneblue.org (/etc/apache2/sites-enabled/vps.zoneblue.org-le-ssl.conf:24)
         port 80 namevhost vps.zoneblue.org (/etc/apache2/sites-enabled/vps.zoneblue.org-le-ssl.conf:43)
         port 80 namevhost vps.zoneblue.org (/etc/apache2/sites-enabled/vps.zoneblue.org-le-ssl.conf:62)
         port 80 namevhost vps.zoneblue.org (/etc/apache2/sites-enabled/vps.zoneblue.org.conf:1)
         port 80 namevhost webmail.zoneblue.org (/etc/apache2/sites-enabled/webmail.zoneblue.org-le-ssl.conf:23)
         port 80 namevhost webmail.zoneblue.org (/etc/apache2/sites-enabled/webmail.zoneblue.org-le-ssl.conf:48)
         port 80 namevhost webmail.zoneblue.org (/etc/apache2/sites-enabled/webmail.zoneblue.org-le-ssl.conf:73)
         port 80 namevhost webmail.zoneblue.org (/etc/apache2/sites-enabled/webmail.zoneblue.org.conf:1)
         port 80 namevhost www.webspaces.net.nz (/etc/apache2/sites-enabled/www.webspaces.net.nz-le-ssl.conf:24)
                 alias webspaces.net.nz
         port 80 namevhost www.webspaces.net.nz (/etc/apache2/sites-enabled/www.webspaces.net.nz.conf:2)
                 alias webspaces.net.nz
         port 80 namevhost www.zoneblue.org (/etc/apache2/sites-enabled/www.zoneblue.org-le-ssl.conf:27)
                 alias zoneblue.org
                 alias zoneblue.nz
                 alias www.zoneblue.nz
         port 80 namevhost www.zoneblue.org (/etc/apache2/sites-enabled/www.zoneblue.org-le-ssl.conf:60)
                 alias zoneblue.org
                 alias zoneblue.nz
                 alias www.zoneblue.nz
         port 80 namevhost www.zoneblue.org (/etc/apache2/sites-enabled/www.zoneblue.org-le-ssl.conf:93)
                 alias zoneblue.org
                 alias zoneblue.nz
                 alias www.zoneblue.nz
         port 80 namevhost www.zoneblue.org (/etc/apache2/sites-enabled/www.zoneblue.org.conf:2)
                 alias zoneblue.org
                 alias zoneblue.nz
                 alias www.zoneblue.nz
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33
2 Likes
vps:/etc/letsencrypt/# cat /home/www/www.zoneblue.org/htdocs/.htaccess

Options +FollowSymLinks  
Options -Indexes  
RewriteEngine On

RewriteRule ^\.well-known\/acme-challenge\/ - [L]

# redirect to https
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteCond %{HTTPS} off
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301,NE]

#redirect the non www variant to the www variant to avoid SEO dup content
RewriteCond %{HTTP_HOST} !^www\.zoneblue\.nz [NC]
RewriteRule (.*) https://www.zoneblue.nz/$1 [R=301,L]

#redirect home page variants to / for same reason
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^/]+/)*index\.php\ HTTP/
RewriteRule ^(([^/]+/)*)index\.php$ https://www.zoneblue.nz/$1 [R=301,L]

#rewrite pretty urls
RewriteCond %{REQUEST_URI} !index\.html$
RewriteRule ^([^/\.]+)\.html$  /cms/page.php?view=$1 [L]

#sitemap
RewriteRule ^sitemap\.xml$ /cms/xml-sitemap.php [L]
2 Likes

The other virtual domains have no htaccess files, nor any DNS cnames.

BTW thanks for looking at this Griffin.

3 Likes

Will return right after lunch

2 Likes

Sorry. Long, unexpected, international call.

What are the outputs of:

sudo certbot certificates
sudo ls -lRa /etc/apache2
sudo cat /etc/apache2/sites-enabled/*.conf > conf.txt

For that last command, please just upload the resulting conf.txt file rather than posting all of the outputs.

2 Likes
vps:/home/peter# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: webspaces.net.nz
    Domains: webspaces.net.nz www.webspaces.net.nz
    Expiry Date: 2022-04-05 01:49:42+00:00 (VALID: 81 days)
    Certificate Path: /etc/letsencrypt/live/webspaces.net.nz/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/webspaces.net.nz/privkey.pem
  Certificate Name: zoneblue.org
    Domains: zoneblue.org map.zoneblue.org vps.zoneblue.org webmail.zoneblue.org                                                                              www.zoneblue.nz www.zoneblue.org zoneblue.nz
    Expiry Date: 2022-04-13 19:42:56+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/zoneblue.org/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/zoneblue.org/privkey.pem
  Certificate Name: thebighouseproject.nz
    Domains: thebighouseproject.nz www.thebighouseproject.nz
    Expiry Date: 2022-04-13 21:04:38+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/thebighouseproject.nz/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/thebighouseproject.nz/privkey.pem
  Certificate Name: circleproject.zoneblue.org
    Domains: circleproject.zoneblue.org
    Expiry Date: 2022-03-01 12:02:07+00:00 (VALID: 46 days)
    Certificate Path: /etc/letsencrypt/live/circleproject.zoneblue.org/fullchain                                                                             .pem
    Private Key Path: /etc/letsencrypt/live/circleproject.zoneblue.org/privkey.p                                                                             em
  Certificate Name: lifeiswonderful.nz
    Domains: lifeiswonderful.nz kinesiology.lifeiswonderful.nz www.lifeiswonderf                                                                             ul.nz
    Expiry Date: 2022-04-05 01:50:09+00:00 (VALID: 81 days)
    Certificate Path: /etc/letsencrypt/live/lifeiswonderful.nz/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/lifeiswonderful.nz/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 Likes
vps:/home/peter# ls -lRa /etc/apache2
/etc/apache2:
total 104
drwxr-xr-x   8 root root  4096 Jan 14 11:05 .
drwxr-xr-x 102 root root  4096 Jan 14 08:40 ..
-rw-r--r--   1 root root  7247 Jul 23 09:09 apache2.conf
-rw-r--r--   1 root root  7114 Nov  5  2016 apache2.conf.dpkg-old
drwxr-xr-x   2 root root  4096 Oct 17 23:38 conf-available
drwxr-xr-x   2 root root  4096 Nov  2  2016 conf-enabled
-rw-r--r--   1 root root  1782 Nov  4  2018 envvars
-rw-r--r--   1 root root 31063 Feb  2  2014 magic
drwxr-xr-x   2 root root 16384 Jan 14 08:40 mods-available
drwxr-xr-x   2 root root  4096 May 14  2019 mods-enabled
-rw-r--r--   1 root root   320 Jul  6  2016 ports.conf
-rw-r--r--   1 root root   772 Aug 25  2016 ports.conf.dpkg-old
drwxr-xr-x   2 root root  4096 Jan 14 11:18 sites-available
drwxr-xr-x   2 root root  4096 Jan 14 11:04 sites-enabled

/etc/apache2/conf-available:
total 28
drwxr-xr-x 2 root root 4096 Oct 17 23:38 .
drwxr-xr-x 8 root root 4096 Jan 14 11:05 ..
-rw-r--r-- 1 root root  315 Jul  6  2016 charset.conf
-rw-r--r-- 1 root root 3224 Jul  6  2016 localized-error-pages.conf
-rw-r--r-- 1 root root  189 Jul  6  2016 other-vhosts-access-log.conf
-rw-r--r-- 1 root root 2174 Nov  4  2018 security.conf
-rw-r--r-- 1 root root  455 Jul  6  2016 serve-cgi-bin.conf

/etc/apache2/conf-enabled:
total 8
drwxr-xr-x 2 root root 4096 Nov  2  2016 .
drwxr-xr-x 8 root root 4096 Jan 14 11:05 ..
lrwxrwxrwx 1 root root   30 Nov  2  2016 charset.conf -> ../conf-available/charset.conf
lrwxrwxrwx 1 root root   44 Nov  2  2016 localized-error-pages.conf -> ../conf-available/localized-error-pages.conf
lrwxrwxrwx 1 root root   46 Nov  2  2016 other-vhosts-access-log.conf -> ../conf-available/other-vhosts-access-log.conf
lrwxrwxrwx 1 root root   31 Nov  2  2016 security.conf -> ../conf-available/security.conf
lrwxrwxrwx 1 root root   36 Nov  2  2016 serve-cgi-bin.conf -> ../conf-available/serve-cgi-bin.conf

/etc/apache2/mods-available:
total 600
drwxr-xr-x 2 root root 16384 Jan 14 08:40 .
drwxr-xr-x 8 root root  4096 Jan 14 11:05 ..
-rw-r--r-- 1 root root   100 Jul  6  2016 access_compat.load
-rw-r--r-- 1 root root   377 Jul  6  2016 actions.conf
-rw-r--r-- 1 root root    66 Nov 15  2009 actions.load
-rw-r--r-- 1 root root   843 Jul  6  2016 alias.conf
-rw-r--r-- 1 root root    62 Nov 15  2009 alias.load
-rw-r--r-- 1 root root    76 Jul  6  2016 allowmethods.load
-rw-r--r-- 1 root root    76 Jul  6  2016 asis.load
-rw-r--r-- 1 root root    94 Jul  6  2016 auth_basic.load
-rw-r--r-- 1 root root    96 Jul  6  2016 auth_digest.load
-rw-r--r-- 1 root root   100 Jul  6  2016 auth_form.load
-rw-r--r-- 1 root root    72 Nov 15  2009 authn_anon.load
-rw-r--r-- 1 root root    72 Jul  6  2016 authn_core.load
-rw-r--r-- 1 root root    85 Nov 15  2009 authn_dbd.load
-rw-r--r-- 1 root root    70 Nov 15  2009 authn_dbm.load
-rw-r--r-- 1 root root    72 Nov 15  2009 authn_file.load
-rw-r--r-- 1 root root    78 Jul  6  2016 authn_socache.load
-rw-r--r-- 1 root root    74 Jul  6  2016 authnz_fcgi.load
-rw-r--r-- 1 root root    90 Nov 15  2009 authnz_ldap.load
-rw-r--r-- 1 root root    72 Jul  6  2016 authz_core.load
-rw-r--r-- 1 root root    96 Jul  6  2016 authz_dbd.load
-rw-r--r-- 1 root root    92 Jul  6  2016 authz_dbm.load
-rw-r--r-- 1 root root   104 Jul  6  2016 authz_groupfile.load
-rw-r--r-- 1 root root    94 Jul  6  2016 authz_host.load
-rw-r--r-- 1 root root    74 Nov 15  2009 authz_owner.load
-rw-r--r-- 1 root root    94 Jul  6  2016 authz_user.load
-rw-r--r-- 1 root root  3374 Jul  6  2016 autoindex.conf
-rw-r--r-- 1 root root    70 Nov 15  2009 autoindex.load
-rw-r--r-- 1 root root    64 Jul  6  2016 buffer.load
-rw-r--r-- 1 root root    62 Nov 15  2009 cache.load
-rw-r--r-- 1 root root   889 Jul  6  2016 cache_disk.conf
-rw-r--r-- 1 root root    89 Jul  6  2016 cache_disk.load
-rw-r--r-- 1 root root    95 Jul  6  2016 cache_socache.load
-rw-r--r-- 1 root root    70 Mar  8  2019 cern_meta.load
-rw-r--r-- 1 root root    58 Nov 15  2009 cgi.load
-rw-r--r-- 1 root root   115 Jul  6  2016 cgid.conf
-rw-r--r-- 1 root root    60 Nov 15  2009 cgid.load
-rw-r--r-- 1 root root    76 Nov 15  2009 charset_lite.load
-rw-r--r-- 1 root root    60 Jul  6  2016 data.load
-rw-r--r-- 1 root root    58 Nov 15  2009 dav.load
-rw-r--r-- 1 root root    83 Jul  6  2016 dav_fs.conf
-rw-r--r-- 1 root root    79 Nov 15  2009 dav_fs.load
-rw-r--r-- 1 root root    68 Nov 15  2009 dav_lock.load
-rw-r--r-- 1 root root  2294 Oct  7  2010 dav_svn.conf
-rw-r--r-- 1 root root   151 Oct  7  2010 dav_svn.load
-rw-r--r-- 1 root root    58 Nov 15  2009 dbd.load
-rw-r--r-- 1 root root   522 Jul  6  2016 deflate.conf
-rw-r--r-- 1 root root    84 Jul  6  2016 deflate.load
-rw-r--r-- 1 root root    64 Jul  6  2016 dialup.load
-rw-r--r-- 1 root root   136 Nov  5  2016 dir.conf
-rw-r--r-- 1 root root    58 Nov 15  2009 dir.load
-rw-r--r-- 1 root root    64 Nov 15  2009 dump_io.load
-rw-r--r-- 1 root root    60 Jul  6  2016 echo.load
-rw-r--r-- 1 root root    58 Nov 15  2009 env.load
-rw-r--r-- 1 root root    66 Nov 15  2009 expires.load
-rw-r--r-- 1 root root    72 Nov 15  2009 ext_filter.load
-rw-r--r-- 1 root root   372 Nov  5  2016 fastcgi.conf
-rw-r--r-- 1 root root    66 Nov 17  2010 fastcgi.load
-rw-r--r-- 1 root root    89 Nov 15  2009 file_cache.load
-rw-r--r-- 1 root root    64 Nov 15  2009 filter.load
-rw-r--r-- 1 root root    66 Nov 15  2009 headers.load
-rw-r--r-- 1 root root   176 Jul  6  2016 heartbeat.load
-rw-r--r-- 1 root root   182 Jul  6  2016 heartmonitor.load
-rw-r--r-- 1 root root    62 Nov  4  2018 http2.load
-rw-r--r-- 1 root root    62 Nov 15  2009 ident.load
-rw-r--r-- 1 root root    68 Mar  8  2019 imagemap.load
-rw-r--r-- 1 root root    82 Jul  6  2016 include.load
-rw-r--r-- 1 root root   402 Jul  6  2016 info.conf
-rw-r--r-- 1 root root    60 Nov 15  2009 info.load
-rw-r--r-- 1 root root   116 Jul  6  2016 lbmethod_bybusyness.load
-rw-r--r-- 1 root root   116 Jul  6  2016 lbmethod_byrequests.load
-rw-r--r-- 1 root root   114 Jul  6  2016 lbmethod_bytraffic.load
-rw-r--r-- 1 root root   114 Jul  6  2016 lbmethod_heartbeat.load
-rw-r--r-- 1 root root   121 Jul  6  2016 ldap.conf
-rw-r--r-- 1 root root    60 Nov 15  2009 ldap.load
-rw-r--r-- 1 root root    70 Jul  6  2016 log_debug.load
-rw-r--r-- 1 root root    76 Nov 15  2009 log_forensic.load
-rw-r--r-- 1 root root    58 Jul  6  2016 lua.load
-rw-r--r-- 1 root root    62 Jul  6  2016 macro.load
-rw-r--r-- 1 root root  7639 Jul  6  2016 mime.conf
-rw-r--r-- 1 root root    60 Nov 15  2009 mime.load
-rw-r--r-- 1 root root   120 Jul  6  2016 mime_magic.conf
-rw-r--r-- 1 root root    72 Nov 15  2009 mime_magic.load
-rw-r--r-- 1 root root   668 Jul  6  2016 mpm_event.conf
-rw-r--r-- 1 root root   106 Jul  6  2016 mpm_event.load
-rw-r--r-- 1 root root   571 Jul  6  2016 mpm_prefork.conf
-rw-r--r-- 1 root root   108 Jul  6  2016 mpm_prefork.load
-rw-r--r-- 1 root root   836 Jul  6  2016 mpm_worker.conf
-rw-r--r-- 1 root root   107 Jul  6  2016 mpm_worker.load
-rw-r--r-- 1 root root   724 Jul  6  2016 negotiation.conf
-rw-r--r-- 1 root root    74 Nov 15  2009 negotiation.load
-rw-r--r-- 1 root root   898 Aug 21  2014 php5.conf
-rw-r--r-- 1 root root    59 Nov 22  2009 php5.load
-rw-r--r-- 1 root root   867 Mar  8  2019 php7.0.conf
-rw-r--r-- 1 root root   102 Mar  8  2019 php7.0.load
-rw-r--r-- 1 root root   822 Jul  6  2016 proxy.conf
-rw-r--r-- 1 root root    62 Nov 15  2009 proxy.load
-rw-r--r-- 1 root root    87 Nov 15  2009 proxy_ajp.load
-rw-r--r-- 1 root root   347 Jul  6  2016 proxy_balancer.conf
-rw-r--r-- 1 root root   115 Jul  6  2016 proxy_balancer.load
-rw-r--r-- 1 root root    95 Nov 15  2009 proxy_connect.load
-rw-r--r-- 1 root root    95 Jul  6  2016 proxy_express.load
-rw-r--r-- 1 root root    89 Jul  6  2016 proxy_fcgi.load
-rw-r--r-- 1 root root    93 Jul  6  2016 proxy_fdpass.load
-rw-r--r-- 1 root root   189 Jul  6  2016 proxy_ftp.conf
-rw-r--r-- 1 root root    87 Nov 15  2009 proxy_ftp.load
-rw-r--r-- 1 root root    93 Nov  4  2018 proxy_hcheck.load
-rw-r--r-- 1 root root  2511 Jul 22  2016 proxy_html.conf
-rw-r--r-- 1 root root    97 Nov  4  2018 proxy_html.load
-rw-r--r-- 1 root root    89 Nov 15  2009 proxy_http.load
-rw-r--r-- 1 root root    97 Nov  4  2018 proxy_http2.load
-rw-r--r-- 1 root root    89 Feb  2  2014 proxy_scgi.load
-rw-r--r-- 1 root root    97 Jul  6  2016 proxy_wstunnel.load
-rw-r--r-- 1 root root    85 Jul  6  2016 ratelimit.load
-rw-r--r-- 1 root root    70 Jul  6  2016 reflector.load
-rw-r--r-- 1 root root    68 Jul  6  2016 remoteip.load
-rw-r--r-- 1 root root  1190 Jul  6  2016 reqtimeout.conf
-rw-r--r-- 1 root root    72 Feb  2  2014 reqtimeout.load
-rw-r--r-- 1 root root    66 Jul  6  2016 request.load
-rw-r--r-- 1 root root    66 Nov 15  2009 rewrite.load
-rw-r--r-- 1 root root    58 Jul  6  2016 sed.load
-rw-r--r-- 1 root root    66 Jul  6  2016 session.load
-rw-r--r-- 1 root root    99 Jul  6  2016 session_cookie.load
-rw-r--r-- 1 root root    99 Jul  6  2016 session_crypto.load
-rw-r--r-- 1 root root    93 Jul  6  2016 session_dbd.load
-rw-r--r-- 1 root root  1280 Jul  6  2016 setenvif.conf
-rw-r--r-- 1 root root    68 Nov 15  2009 setenvif.load
-rw-r--r-- 1 root root    78 Jul  6  2016 slotmem_plain.load
-rw-r--r-- 1 root root    74 Jul  6  2016 slotmem_shm.load
-rw-r--r-- 1 root root    74 Jul  6  2016 socache_dbm.load
-rw-r--r-- 1 root root    84 Jul  6  2016 socache_memcache.load
-rw-r--r-- 1 root root    78 Jul  6  2016 socache_shmcb.load
-rw-r--r-- 1 root root    66 Nov 15  2009 speling.load
-rw-r--r-- 1 root root  3110 Jul  6  2016 ssl.conf
-rw-r--r-- 1 root root    97 Jul  6  2016 ssl.load
-rw-r--r-- 1 root root   749 Jul  6  2016 status.conf
-rw-r--r-- 1 root root    64 Nov 15  2009 status.load
-rw-r--r-- 1 root root    72 Nov 15  2009 substitute.load
-rw-r--r-- 1 root root    64 Nov 15  2009 suexec.load
-rw-r--r-- 1 root root    70 Nov 15  2009 unique_id.load
-rw-r--r-- 1 root root   324 Nov  4  2018 userdir.conf
-rw-r--r-- 1 root root    66 Nov 15  2009 userdir.load
-rw-r--r-- 1 root root    70 Nov 15  2009 usertrack.load
-rw-r--r-- 1 root root    74 Nov 15  2009 vhost_alias.load
-rw-r--r-- 1 root root    66 Jul  6  2016 xml2enc.load

/etc/apache2/mods-enabled:
total 12
drwxr-xr-x 2 root root 4096 May 14  2019 .
drwxr-xr-x 8 root root 4096 Jan 14 11:05 ..
lrwxrwxrwx 1 root root   36 Nov  2  2016 access_compat.load -> ../mods-available/access_compat.load
lrwxrwxrwx 1 root root   30 Oct  3  2014 actions.conf -> ../mods-available/actions.conf
lrwxrwxrwx 1 root root   30 Oct  3  2014 actions.load -> ../mods-available/actions.load
lrwxrwxrwx 1 root root   28 Apr  5  2010 alias.conf -> ../mods-available/alias.conf
lrwxrwxrwx 1 root root   28 Apr  5  2010 alias.load -> ../mods-available/alias.load
lrwxrwxrwx 1 root root   33 Apr  5  2010 auth_basic.load -> ../mods-available/auth_basic.load
lrwxrwxrwx 1 root root   33 Nov  2  2016 authn_core.load -> ../mods-available/authn_core.load
lrwxrwxrwx 1 root root   33 Apr  5  2010 authn_file.load -> ../mods-available/authn_file.load
lrwxrwxrwx 1 root root   33 Nov  2  2016 authz_core.load -> ../mods-available/authz_core.load
lrwxrwxrwx 1 root root   38 Apr  5  2010 authz_groupfile.load -> ../mods-available/authz_groupfile.load
lrwxrwxrwx 1 root root   33 Apr  5  2010 authz_host.load -> ../mods-available/authz_host.load
lrwxrwxrwx 1 root root   33 Apr  5  2010 authz_user.load -> ../mods-available/authz_user.load
lrwxrwxrwx 1 root root   32 Apr  5  2010 autoindex.conf -> ../mods-available/autoindex.conf
lrwxrwxrwx 1 root root   32 Apr  5  2010 autoindex.load -> ../mods-available/autoindex.load
lrwxrwxrwx 1 root root   26 Apr  5  2010 cgi.load -> ../mods-available/cgi.load
lrwxrwxrwx 1 root root   27 Oct  3  2014 cgid.conf -> ../mods-available/cgid.conf
lrwxrwxrwx 1 root root   27 Oct  3  2014 cgid.load -> ../mods-available/cgid.load
lrwxrwxrwx 1 root root   26 Mar  1  2011 dav.load -> ../mods-available/dav.load
lrwxrwxrwx 1 root root   30 Apr  5  2010 deflate.conf -> ../mods-available/deflate.conf
lrwxrwxrwx 1 root root   30 Apr  5  2010 deflate.load -> ../mods-available/deflate.load
lrwxrwxrwx 1 root root   26 Apr  5  2010 dir.conf -> ../mods-available/dir.conf
lrwxrwxrwx 1 root root   26 Apr  5  2010 dir.load -> ../mods-available/dir.load
lrwxrwxrwx 1 root root   26 Apr  5  2010 env.load -> ../mods-available/env.load
lrwxrwxrwx 1 root root   30 Nov  5  2016 fastcgi.conf -> ../mods-available/fastcgi.conf
-rw-r--r-- 1 root root  436 Nov  5  2016 fastcgi.conf.backup
lrwxrwxrwx 1 root root   30 Nov  5  2016 fastcgi.load -> ../mods-available/fastcgi.load
lrwxrwxrwx 1 root root   29 Nov  2  2016 filter.load -> ../mods-available/filter.load
lrwxrwxrwx 1 root root   27 Apr  5  2010 mime.conf -> ../mods-available/mime.conf
lrwxrwxrwx 1 root root   27 Apr  5  2010 mime.load -> ../mods-available/mime.load
lrwxrwxrwx 1 root root   34 May 14  2019 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf
lrwxrwxrwx 1 root root   34 May 14  2019 mpm_prefork.load -> ../mods-available/mpm_prefork.load
lrwxrwxrwx 1 root root   34 Apr  5  2010 negotiation.conf -> ../mods-available/negotiation.conf
lrwxrwxrwx 1 root root   34 Apr  5  2010 negotiation.load -> ../mods-available/negotiation.load
lrwxrwxrwx 1 root root   29 May 14  2019 php7.0.conf -> ../mods-available/php7.0.conf
lrwxrwxrwx 1 root root   29 May 14  2019 php7.0.load -> ../mods-available/php7.0.load
lrwxrwxrwx 1 root root   33 Oct  2  2014 reqtimeout.conf -> ../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root   33 Oct  2  2014 reqtimeout.load -> ../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root   30 Mar 31  2014 rewrite.load -> ../mods-available/rewrite.load
lrwxrwxrwx 1 root root   31 Apr  5  2010 setenvif.conf -> ../mods-available/setenvif.conf
lrwxrwxrwx 1 root root   31 Apr  5  2010 setenvif.load -> ../mods-available/setenvif.load
lrwxrwxrwx 1 root root   36 Nov  2  2016 socache_shmcb.load -> ../mods-available/socache_shmcb.load
lrwxrwxrwx 1 root root   26 Aug 25  2016 ssl.conf -> ../mods-available/ssl.conf
lrwxrwxrwx 1 root root   26 Aug 25  2016 ssl.load -> ../mods-available/ssl.load
lrwxrwxrwx 1 root root   29 Apr  5  2010 status.conf -> ../mods-available/status.conf
lrwxrwxrwx 1 root root   29 Apr  5  2010 status.load -> ../mods-available/status.load

/etc/apache2/sites-available:
total 96
drwxr-xr-x 2 root root 4096 Jan 14 11:18 .
drwxr-xr-x 8 root root 4096 Jan 14 11:05 ..
-rw-r--r-- 1 root root  507 Nov  5  2016 000-default.conf
-rw-r--r-- 1 root root  806 Oct  2 20:09 circleproject.zoneblue.org-le-ssl.conf
-rw-r--r-- 1 root root  557 Oct  2 18:17 circleproject.zoneblue.org.conf
-rw-r--r-- 1 root root 6338 Nov  4  2018 default-ssl.conf
-rw-r--r-- 1 root root  811 May  9  2021 kinesiology.lifeiswonderful.nz-le-ssl.conf
-rw-r--r-- 1 root root  578 Jan  9  2021 kinesiology.lifeiswonderful.nz.conf
-rw-r--r-- 1 root root  576 Jul 23 13:58 lifeiswonderful.co.nz.conf
-rw-r--r-- 1 root root  791 May  9  2021 lifeiswonderful.nz-le-ssl.conf
-rw-r--r-- 1 root root  750 Feb 15  2020 lifeiswonderful.nz.conf
-rw-r--r-- 1 root root 2891 Jan 14 09:43 map.zoneblue.org-le-ssl.conf
-rw-r--r-- 1 root root  507 Nov  5  2016 map.zoneblue.org.conf
-rw-r--r-- 1 root root  810 Jan 14 11:18 thebighouseproject.nz-le-ssl.conf
-rw-r--r-- 1 root root  585 Jan 14 11:17 thebighouseproject.nz.conf
-rw-r--r-- 1 root root 2335 Jan 14 09:43 vps.zoneblue.org-le-ssl.conf
-rw-r--r-- 1 root root  503 Mar  1  2021 vps.zoneblue.org.conf
-rw-r--r-- 1 root root 3252 Jan 14 09:43 webmail.zoneblue.org-le-ssl.conf
-rw-r--r-- 1 root root  665 Feb 15  2020 webmail.zoneblue.org.conf
-rw-r--r-- 1 root root 1717 May  9  2021 www.webspaces.net.nz-le-ssl.conf
-rw-r--r-- 1 root root  753 Feb 15  2020 www.webspaces.net.nz.conf
-rw-r--r-- 1 root root 3977 Jan 14 09:43 www.zoneblue.org-le-ssl.conf
-rw-r--r-- 1 root root  877 Feb 15  2020 www.zoneblue.org.conf

/etc/apache2/sites-enabled:
total 24
drwxr-xr-x 2 root root 4096 Jan 14 11:04 .
drwxr-xr-x 8 root root 4096 Jan 14 11:05 ..
lrwxrwxrwx 1 root root   35 Nov  5  2016 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root root   71 Mar  1  2021 kinesiology.lifeiswonderful.nz-le-ssl.conf -> /etc/apache2/sites-available/kinesiology.lifeiswonderful.nz-le-ssl.conf
lrwxrwxrwx 1 root root   54 Jan  9  2021 kinesiology.lifeiswonderful.nz.conf -> ../sites-available/kinesiology.lifeiswonderful.nz.conf
lrwxrwxrwx 1 root root   59 Feb 15  2020 lifeiswonderful.nz-le-ssl.conf -> /etc/apache2/sites-available/lifeiswonderful.nz-le-ssl.conf
lrwxrwxrwx 1 root root   42 Feb 15  2020 lifeiswonderful.nz.conf -> ../sites-available/lifeiswonderful.nz.conf
lrwxrwxrwx 1 root root   57 Jan 14 09:43 map.zoneblue.org-le-ssl.conf -> /etc/apache2/sites-available/map.zoneblue.org-le-ssl.conf
lrwxrwxrwx 1 root root   40 Nov  5  2016 map.zoneblue.org.conf -> ../sites-available/map.zoneblue.org.conf
lrwxrwxrwx 1 root root   62 Jan 14 11:04 thebighouseproject.nz-le-ssl.conf -> /etc/apache2/sites-available/thebighouseproject.nz-le-ssl.conf
lrwxrwxrwx 1 root root   45 Jan  3 07:38 thebighouseproject.nz.conf -> ../sites-available/thebighouseproject.nz.conf
lrwxrwxrwx 1 root root   57 Jan 14 09:43 vps.zoneblue.org-le-ssl.conf -> /etc/apache2/sites-available/vps.zoneblue.org-le-ssl.conf
lrwxrwxrwx 1 root root   40 Mar  1  2021 vps.zoneblue.org.conf -> ../sites-available/vps.zoneblue.org.conf
lrwxrwxrwx 1 root root   61 Jan 14 09:43 webmail.zoneblue.org-le-ssl.conf -> /etc/apache2/sites-available/webmail.zoneblue.org-le-ssl.conf
lrwxrwxrwx 1 root root   44 Nov  5  2016 webmail.zoneblue.org.conf -> ../sites-available/webmail.zoneblue.org.conf
lrwxrwxrwx 1 root root   61 May  9  2021 www.webspaces.net.nz-le-ssl.conf -> /etc/apache2/sites-available/www.webspaces.net.nz-le-ssl.conf
lrwxrwxrwx 1 root root   44 Nov  5  2016 www.webspaces.net.nz.conf -> ../sites-available/www.webspaces.net.nz.conf
lrwxrwxrwx 1 root root   57 Jan 14 09:42 www.zoneblue.org-le-ssl.conf -> /etc/apache2/sites-available/www.zoneblue.org-le-ssl.conf
lrwxrwxrwx 1 root root   40 Nov  5  2016 www.zoneblue.org.conf -> ../sites-available/www.zoneblue.org.conf
2 Likes

https://zoneblue.org/files/apacheconf.txt

3 Likes

I will note that i use this same strategy on the other virtual hosts on that box, without issue. This is, though, the more involved of the certs, for sure.

2 Likes

Seeing the same name multiple times is concerning:

Some of which are in the same file (some are in multiple files - even more concerning).

3 Likes

Might this have something todo with my every 3 months delete and new cert? I only figured out there was a delete command line function in the last loop. Before that i deleted entries in archive, live and renewal.

2 Likes

You shouldn't need to delete anything and you especially shouldn't be manually deleting anything. Give me a bit to look things over.

2 Likes

There are certainly numerous problems. I'm working on correcting them. This will take a bit.

3 Likes

Sure no hurry, been that way for a year or more. Does get a bit stressful on day 89 with impending no web no email :wink: Hence the brute force.

2 Likes

Alright...

Run these:

sudo a2dissite *.conf
sudo mkdir /etc/apache2/sites-available/old
sudo mv /etc/apache2/sites-available/*.conf /etc/apache2/sites-available/old

sudo certbot delete --cert-name webspaces.net.nz
sudo certbot delete --cert-name zoneblue.org
sudo certbot delete --cert-name thebighouseproject.nz
sudo certbot delete --cert-name circleproject.zoneblue.org
sudo certbot delete --cert-name lifeiswonderful.nz

Download these into /etc/apache2/sites-available :

www.zoneblue.org.conf.txt (563 Bytes)
www.webspaces.net.nz.conf.txt (563 Bytes)
webmail.zoneblue.org.conf.txt (525 Bytes)
vps.zoneblue.org.conf.txt (500 Bytes)
thebighouseproject.nz.conf.txt (583 Bytes)
map.zoneblue.org.conf.txt (505 Bytes)
lifeiswonderful.nz.conf.txt (556 Bytes)
kinesiology.lifeiswonderful.nz.conf.txt (576 Bytes)

Run these:

sudo mv /etc/apache2/sites-available/www.zoneblue.org.conf.txt /etc/apache2/sites-available/www.zoneblue.org.conf
sudo mv /etc/apache2/sites-available/www.webspaces.net.nz.conf.txt /etc/apache2/sites-available/www.webspaces.net.nz.conf
sudo mv /etc/apache2/sites-available/webmail.zoneblue.org.conf.txt etc/apache2/sites-available/webmail.zoneblue.org.conf
sudo mv /etc/apache2/sites-available/vps.zoneblue.org.conf.txt /etc/apache2/sites-available/vps.zoneblue.org.conf
sudo mv /etc/apache2/sites-available/thebighouseproject.nz.conf.txt /etc/apache2/sites-available/thebighouseproject.nz.conf
sudo mv /etc/apache2/sites-available/map.zoneblue.org.conf.txt /etc/apache2/sites-available/map.zoneblue.org.conf
sudo mv /etc/apache2/sites-available/lifeiswonderful.nz.conf.txt /etc/apache2/sites-available/lifeiswonderful.nz.conf
sudo mv /etc/apache2/sites-available/kinesiology.lifeiswonderful.nz.conf.txt /etc/apache2/sites-available/kinesiology.lifeiswonderful.nz.conf

sudo a2ensite *.conf
sudo apachectl -k graceful

sudo certbot --apache -d "www.zoneblue.org,zoneblue.org,www.zoneblue.nz,zoneblue.nz"
sudo certbot --apache -d "www.webspaces.net.nz,webspaces.net.nz"
sudo certbot --apache -d "webmail.zoneblue.org"
sudo certbot --apache -d "vps.zoneblue.org"
sudo certbot --apache -d "thebighouseproject.nz,www.thebighouseproject.nz"
sudo certbot --apache -d "map.zoneblue.org"
sudo certbot --apache -d "lifeiswonderful.nz,www.lifeiswonderful.nz"
sudo certbot --apache -d "kinesiology.lifeiswonderful.nz"
2 Likes

Ok i see what you are doing here: clean out the certs and the virtual hosts, and rebuild from scratch. Here is the afterwards apacheconf. Looks much cleaner.

https://zoneblue.org/files/afterapacheconf.txt

So, i happened to notice something yesterday which possibly explains all this.
Certbot is adding these duplicate virtual host records of the form:

vps:/etc/apache2/sites-available/old# cat map.zoneblue.org-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
     ServerAdmin webmaster@zoneblue.org
     ServerName  map.zoneblue.org
     DocumentRoot /home/www/map.zoneblue.org/htdocs/
     ErrorLog  /home/www/map.zoneblue.org/logs/error.log
     CustomLog /home/www/map.zoneblue.org/logs/access.log combined

     <Directory  /home/www/map.zoneblue.org/htdocs/>
        Options Indexes FollowSymLinks MultiViews
        Require all granted
        AllowOverride All
        DirectoryIndex index.php index.html
     </Directory>

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/zoneblue.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/zoneblue.org/privkey.pem
</VirtualHost>

</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
     ServerAdmin webmaster@zoneblue.org
     ServerName  map.zoneblue.org
     DocumentRoot /home/www/map.zoneblue.org/htdocs/
     ErrorLog  /home/www/map.zoneblue.org/logs/error.log
     CustomLog /home/www/map.zoneblue.org/logs/access.log combined

     <Directory  /home/www/map.zoneblue.org/htdocs/>
        Options Indexes FollowSymLinks MultiViews
        Require all granted
        AllowOverride All
        DirectoryIndex index.php index.html
     </Directory>

</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
     ServerAdmin webmaster@zoneblue.org
     ServerName  map.zoneblue.org
     DocumentRoot /home/www/map.zoneblue.org/htdocs/
     ErrorLog  /home/www/map.zoneblue.org/logs/error.log
     CustomLog /home/www/map.zoneblue.org/logs/access.log combined

     <Directory  /home/www/map.zoneblue.org/htdocs/>
        Options Indexes FollowSymLinks MultiViews
        Require all granted
        AllowOverride All
        DirectoryIndex index.php index.html
     </Directory>

</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
     ServerAdmin webmaster@zoneblue.org
     ServerName  map.zoneblue.org
     DocumentRoot /home/www/map.zoneblue.org/htdocs/
     ErrorLog  /home/www/map.zoneblue.org/logs/error.log
     CustomLog /home/www/map.zoneblue.org/logs/access.log combined

     <Directory  /home/www/map.zoneblue.org/htdocs/>
        Options Indexes FollowSymLinks MultiViews
        Require all granted
        AllowOverride All
        DirectoryIndex index.php index.html
     </Directory>

</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
     ServerAdmin webmaster@zoneblue.org
     ServerName  map.zoneblue.org
     DocumentRoot /home/www/map.zoneblue.org/htdocs/
     ErrorLog  /home/www/map.zoneblue.org/logs/error.log
     CustomLog /home/www/map.zoneblue.org/logs/access.log combined

     <Directory  /home/www/map.zoneblue.org/htdocs/>
        Options Indexes FollowSymLinks MultiViews
        Require all granted
        AllowOverride All
        DirectoryIndex index.php index.html
     </Directory>
</VirtualHost>
</IfModule>

By chance yesterday i observed this at work.

When i last removed the zoneblue org (BTW these cert names are tripping your spambot) cert and got a new one, the virtual sites-available record for thebighouseproject nz (certbot version) disappeared. (i dont have an explanation for this, it may be related to the quagmire inside the virtual host records). Anyway, certbot certificates showed that the bighouseproject cert still existed, so reading the certbot manual i couldnt find a command to just use the existing cert and to rebuild the sites-available record, so i rebuilt it by hand, which didnt satisfy the web browser, so i used certbot delete and got a new cert, and new virtual host record. (But without disabling the certbot host record or cleaning up the sites-available records. (thinking it would just overwrite the old record).
So what it actually did, is yes it did overwrite the certbot bighouseproject virtual host record but it contained both the new certbot secure section and the port 80 version copied from the non certbot version. Following? Somehow with my messy (incomplete understanding of how to cleanly remove certs) workflow, certbot is adding port 80 sections to the certbot sites-available records, which then become a duplicate of the existing port 80 record. I don't have an explanation why certbot does this. Maybe you guys will understand it.

For my part i now know, to keep sites-available clean, in order to remove a cert, i need to a2dissite the certbot version of the virtual host, then use certbot delete. Lesson learned, and thanks so much for you extensive assistance Griffin.

1 Like

You are quite welcome!

:blush:

This unfortunate duplication issue is known with certbot. I think it was addressed in a newer version. The version you have (0.28.0) qualifies as ancient at this point.

There shouldn't ever be port 80 vHosts with "if SSL" wrappers around them.

I took a look through your new apache configuration dump. Looks good! :+1:

2 Likes

Tagging @certbot-devs for awareness of this situation and comment if desired.

3 Likes