Weird issue with mutli domain certificate

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
hekaya.nl

moredhel.org

I ran this command:
I used the acme.sh client to revoke my multi domain wildcard certificate and generated a new certificate with an extra domain (hekaya.nl). I have left out a few domains here to keep the command readable: ./acme.sh --server letsencrypt --issue --dns dns_nsupdate --dnssleep 120 --ocsp -d 'valheru.org' -d '.valheru.org' -d 'moredhel.org' -d '.moredhel.org' -d 'hekaya.nl' -d '*.hekaya.nl'

It produced this output:
I have what seems to be a valid certificate, ssllabs.com, internet.nl and digicert.com all vind my certificate valid for my domains EXCEPT for hekaya.nl. Note, I use this certificate in the same vhost in Apache so all settings are exactly the same. On hekaya.nl I get a chain error on ssllabs.com and internet.nl and an oscp stapling error on digicert.com.
I have been using this setup for about a year and never had issues like this. My guess is that I am missing something minor but I can't figure out what at the moment.

My web server is (include version):
Server version: Apache/2.4.52 (Ubuntu)
Server built: 2024-07-17T18:57:26

The operating system my web server runs on is (include version):
Ubuntu 22.04

My hosting provider, if applicable, is:
Private VPS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

v3.0.8

Welcome @valheru

The 3 domain names I checked all use the wrong cert and chain. There were different problems for different domains

valheru.org sends the cert leaf twice https://decoder.link/sslchecker/valheru.org/443
moredhel.org only sends the leaf and no chain https://decoder.link/sslchecker/moredhel.org/443
hekaya.nl also only sends the leaf

We need to review your Apache config. This is most likely what is wrong. Please show output of

sudo apache2ctl -t -D DUMP_VHOSTS
4 Likes

This is the complete dump:

VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server vps01.valheru.org (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost vps01.valheru.org (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost autodiscover.valheru.org (/etc/apache2/sites-enabled/001-autodiscover.valheru.org.conf:1)
                 alias autodiscover.beijum-nieuws.nl
                 alias autodiscover.bentismaheerd.nl
                 alias autodiscover.deprinsenhoek.nl
                 alias autodiscover.drlnet.com
                 alias autodiscover.drlnet.net
                 alias autodiscover.eledhel.org
                 alias autodiscover.glamredhel.org
                 alias autodiscover.kyuubi.nl
                 alias autodiscover.moredhel.org
                 alias autodiscover.nodig.nu
                 alias autodiscover.taredhel.org
                 alias autoconfig.beijum-nieuws.nl
                 alias autoconfig.bentismaheerd.nl
                 alias autoconfig.deprinsenhoek.nl
                 alias autoconfig.drlnet.com
                 alias autoconfig.drlnet.net
                 alias autoconfig.eledhel.org
                 alias autoconfig.glamredhel.org
                 alias autoconfig.kyuubi.nl
                 alias autoconfig.moredhel.org
                 alias autoconfig.nodig.nu
                 alias autoconfig.taredhel.org
                 alias autoconfig.valheru.org
         port 80 namevhost autodiscover.beijum-nieuws.nl (/etc/apache2/sites-enabled/001-autodiscover.valheru.org.conf:73)
                 alias autoconfig.beijum-nieuws.nl
         port 80 namevhost calendar.valheru.org (/etc/apache2/sites-enabled/001-calendar.valheru.org.conf:1)
                 alias contacts.valheru.org
                 alias radicale.valheru.org
         port 80 namevhost lists.valheru.org (/etc/apache2/sites-enabled/001-lists.valheru.org.conf:1)
         port 80 namevhost matomo.valheru.org (/etc/apache2/sites-enabled/001-matomo.valheru.org.conf:5)
         port 80 namevhost phpmyadmin.valheru.org (/etc/apache2/sites-enabled/001-phpmyadmin.valheru.org.conf:4)
         port 80 namevhost postfixadmin.valheru.org (/etc/apache2/sites-enabled/001-postfixadmin.valheru.org.conf:1)
         port 80 namevhost rspamd.valheru.org (/etc/apache2/sites-enabled/001-rspamd.valheru.org.conf:1)
         port 80 namevhost webdav.valheru.org (/etc/apache2/sites-enabled/001-webdav.valheru.org.conf:7)
         port 80 namevhost webmail.valheru.org (/etc/apache2/sites-enabled/001-webmail.valheru.org.conf:5)
         port 80 namevhost csp-reporter.valheru.org (/etc/apache2/sites-enabled/005-csp-reporter.valheru.org.conf:5)
         port 80 namevhost mta-sts.valheru.org (/etc/apache2/sites-enabled/005-mta-sts.valheru.org.conf:1)
                 alias mta-sts.bentismaheerd.nl
                 alias mta-sts.deprinsenhoek.nl
                 alias mta-sts.drlnet.com
                 alias mta-sts.drlnet.net
                 alias mta-sts.eledhel.org
                 alias mta-sts.glamredhel.org
                 alias mta-sts.kremersheerd.nl
                 alias mta-sts.kyuubi.nl
                 alias mta-sts.moredhel.org
                 alias mta-sts.nodig.nu
                 alias mta-sts.taredhel.org
                 alias mta-sts.hekaya.nl
         port 80 namevhost mta-sts.beijum-nieuws.nl (/etc/apache2/sites-enabled/005-mta-sts.valheru.org.conf:67)
         port 80 namevhost bentismaheerd.nl (/etc/apache2/sites-enabled/010-bentismaheerd.nl.conf:2)
                 wild alias *.bentismaheerd.nl
         port 80 namevhost drlnet.com (/etc/apache2/sites-enabled/010-drlnet.com.conf:2)
                 wild alias *.drlnet.com
         port 80 namevhost nodig.nu (/etc/apache2/sites-enabled/010-nodig.nu.conf:2)
                 wild alias *.nodig.nu
         port 80 namevhost valheru.org (/etc/apache2/sites-enabled/010-valheru.org.conf:2)
                 alias moredhel.org
                 alias eledhel.org
                 alias glamredhel.org
                 alias taredhel.org
                 alias hekaya.nl
                 alias kyuubi.nl
                 wild alias *.moredhel.org
                 wild alias *.eledhel.org
                 wild alias *.glamredhel.org
                 wild alias *.taredhel.org
                 wild alias *.hekaya.nl
                 wild alias *.kyuubi.nl
                 wild alias *.valheru.org
         port 80 namevhost beijum-nieuws.nl (/etc/apache2/sites-enabled/011-beijum-nieuws.nl.conf:1)
                 wild alias *.beijum-nieuws.nl
*:443                  is a NameVirtualHost
         default server vps01.valheru.org (/etc/apache2/sites-enabled/000-default-ssl.conf:2)
         port 443 namevhost vps01.valheru.org (/etc/apache2/sites-enabled/000-default-ssl.conf:2)
         port 443 namevhost autodiscover.valheru.org (/etc/apache2/sites-enabled/001-autodiscover.valheru.org.conf:31)
                 alias autodiscover.beijum-nieuws.nl
                 alias autodiscover.bentismaheerd.nl
                 alias autodiscover.deprinsenhoek.nl
                 alias autodiscover.drlnet.com
                 alias autodiscover.drlnet.net
                 alias autodiscover.eledhel.org
                 alias autodiscover.glamredhel.org
                 alias autodiscover.kyuubi.nl
                 alias autodiscover.moredhel.org
                 alias autodiscover.nodig.nu
                 alias autodiscover.taredhel.org
                 alias autoconfig.beijum-nieuws.nl
                 alias autoconfig.bentismaheerd.nl
                 alias autoconfig.deprinsenhoek.nl
                 alias autoconfig.drlnet.com
                 alias autoconfig.drlnet.net
                 alias autoconfig.eledhel.org
                 alias autoconfig.glamredhel.org
                 alias autoconfig.kyuubi.nl
                 alias autoconfig.moredhel.org
                 alias autoconfig.nodig.nu
                 alias autoconfig.taredhel.org
                 alias autoconfig.valheru.org
         port 443 namevhost autodiscover.beijum-nieuws.nl (/etc/apache2/sites-enabled/001-autodiscover.valheru.org.conf:101)
                 alias autoconfig.beijum-nieuws.nl
         port 443 namevhost calendar.valheru.org (/etc/apache2/sites-enabled/001-calendar.valheru.org.conf:23)
                 alias contacts.valheru.org
                 alias radicale.valheru.org
         port 443 namevhost lists.valheru.org (/etc/apache2/sites-enabled/001-lists.valheru.org.conf:23)
         port 443 namevhost matomo.valheru.org (/etc/apache2/sites-enabled/001-matomo.valheru.org.conf:26)
         port 443 namevhost phpmyadmin.valheru.org (/etc/apache2/sites-enabled/001-phpmyadmin.valheru.org.conf:25)
         port 443 namevhost postfixadmin.valheru.org (/etc/apache2/sites-enabled/001-postfixadmin.valheru.org.conf:22)
         port 443 namevhost rspamd.valheru.org (/etc/apache2/sites-enabled/001-rspamd.valheru.org.conf:22)
         port 443 namevhost webdav.valheru.org (/etc/apache2/sites-enabled/001-webdav.valheru.org.conf:28)
         port 443 namevhost webmail.valheru.org (/etc/apache2/sites-enabled/001-webmail.valheru.org.conf:26)
         port 443 namevhost csp-reporter.valheru.org (/etc/apache2/sites-enabled/005-csp-reporter.valheru.org.conf:26)
         port 443 namevhost mta-sts.valheru.org (/etc/apache2/sites-enabled/005-mta-sts.valheru.org.conf:23)
                 alias mta-sts.bentismaheerd.nl
                 alias mta-sts.deprinsenhoek.nl
                 alias mta-sts.drlnet.com
                 alias mta-sts.drlnet.net
                 alias mta-sts.eledhel.org
                 alias mta-sts.glamredhel.org
                 alias mta-sts.kremersheerd.nl
                 alias mta-sts.kyuubi.nl
                 alias mta-sts.moredhel.org
                 alias mta-sts.nodig.nu
                 alias mta-sts.taredhel.org
                 alias mta-sts.hekaya.nl
         port 443 namevhost mta-sts.beijum-nieuws.nl (/etc/apache2/sites-enabled/005-mta-sts.valheru.org.conf:88)
         port 443 namevhost bentismaheerd.nl (/etc/apache2/sites-enabled/010-bentismaheerd.nl.conf:27)
                 wild alias *.bentismaheerd.nl
         port 443 namevhost drlnet.com (/etc/apache2/sites-enabled/010-drlnet.com.conf:27)
                 wild alias *.drlnet.com
         port 443 namevhost www.nodig.nu (/etc/apache2/sites-enabled/010-nodig.nu.conf:28)
                 wild alias *.nodig.nu
         port 443 namevhost nodig.nu (/etc/apache2/sites-enabled/010-nodig.nu.conf:69)
                 alias embed.nodig.nu
         port 443 namevhost moredhel.org (/etc/apache2/sites-enabled/010-valheru.org.conf:33)
                 alias eledhel.org
                 alias glamredhel.org
                 alias taredhel.org
                 alias hekaya.nl
                 alias kyuubi.nl
                 wild alias *.eledhel.org
                 wild alias *.glamredhel.org
                 wild alias *.taredhel.org
                 wild alias *.valheru.org
                 wild alias *.moredhel.org
                 wild alias *.hekaya.nl
                 wild alias *.kyuubi.nl
         port 443 namevhost valheru.org (/etc/apache2/sites-enabled/010-valheru.org.conf:78)
                 alias embed.valheru.org
         port 443 namevhost beijum-nieuws.nl (/etc/apache2/sites-enabled/011-beijum-nieuws.nl.conf:23)
                 wild alias *.beijum-nieuws.nl

Let's review them one at a time. Hopefully this will show the pattern to the problem.

Please show contents of above file which is VirtualHost for apex name valheru.org

3 Likes

This is the file:

root@vps01:/etc/apache2/sites-enabled# cat 010-valheru.org.conf
# Redirect valheru.org to https
<VirtualHost *:80>
    # Actual sites
    ServerName valheru.org

    # Aliasses
    ServerAlias moredhel.org *.moredhel.org
    ServerAlias eledhel.org *.eledhel.org
    ServerAlias glamredhel.org *.glamredhel.org
    ServerAlias taredhel.org *.taredhel.org
    ServerAlias hekaya.nl *.hekaya.nl
    ServerAlias kyuubi.nl *.kyuubi.nl
    ServerAlias *.valheru.org

    RewriteEngine on
    RewriteCond %{REQUEST_URI} !\.well-known/acme-challenge
    RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

    Documentroot /var/www/html
    <Directory /var/www/html>
        Options +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    CustomLog "/var/log/apache2/valheru.org-ssl-access.log" combined
    ErrorLog "/var/log/apache2/valheru.org-ssl-error.log"
    LogLevel debug

</VirtualHost>

# Redirect to valheru.org
<VirtualHost *:443>
    # Actual sites
    ServerName moredhel.org

    # Aliasses
    ServerAlias eledhel.org *.eledhel.org
    ServerAlias glamredhel.org *.glamredhel.org
    ServerAlias taredhel.org *.taredhel.org
    ServerAlias *.valheru.org *.moredhel.org
    ServerAlias hekaya.nl *.hekaya.nl
    ServerAlias kyuubi.nl *.kyuubi.nl

    # SSL settings
    SSLEngine on
    SSLUseStapling on
    SSLCertificateFile "/etc/letsencrypt/live/valheru.org/cert.pem"
    SSLCertificateChainFile "/etc/letsencrypt/live/valheru.org/chain.pem"
    SSLCertificateKeyFile "/etc/letsencrypt/live/valheru.org/privkey.pem"

    # Header settings
    #Header unset Server
    Header always set X-Frame-Options SAMEORIGIN
    Header always set X-Xss-Protection "1; mode=block"
    Header always set X-Content-Type-Options "nosniff"
    Header always set Content-Security-Policy "default-src https: data: 'unsafe-inline' 'unsafe-eval'"
    Header always set Strict-Transport-Security "max-age=31536000;"

    RewriteEngine on
    RewriteCond %{REQUEST_URI} !\.well-known/acme-challenge
    #RewriteCond %{REQUEST_URI} !\.well-known/autoconfig
    RewriteRule ^(.*)$ https://valheru.org%{REQUEST_URI} [L,R=301]

    Documentroot /var/www/html
    <Directory /var/www/html>
        Options +FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    CustomLog "/var/log/apache2/valheru.org-ssl-access.log" combined
    ErrorLog "/var/log/apache2/valheru.org-ssl-error.log"
    LogLevel debug

</VirtualHost>

<VirtualHost *:443>
    # Actual site
    ServerName valheru.org

    # Aliasses
    ServerAlias embed.valheru.org

    # SSL settings
    SSLEngine on
    SSLUseStapling on
    SSLCertificateFile          /etc/letsencrypt/live/valheru.org/cert.pem
    SSLCertificateKeyFile       /etc/letsencrypt/live/valheru.org/privkey.pem
    SSLCertificateChainFile     /etc/letsencrypt/live/valheru.org/fullchain.pem

    # Header settings
    #Header unset Server
    Header always set X-Frame-Options SAMEORIGIN
    Header always set X-Xss-Protection "1; mode=block"
    Header always set X-Content-Type-Options "nosniff"
    Header always set Content-Security-Policy: "default-src 'self' valheru.org www.valheru.org moredhel.org; \
                        script-src              'self' 'unsafe-inline' 'unsafe-eval' valheru.org moredhel.org static.addtoany.com; \
                        script-src-elem         'self' 'unsafe-inline' valheru.org www.valheru.org moredhel.org static.addtoany.com connect.facebook.net player.vimeo.com; \
                        script-src-attr         'self' 'unsafe-inline' valheru.org www.valheru.org moredhel.org fonts.gstatic.com fonts.googleapis.com player.vimeo.com; \
                        object-src              'self' 'unsafe-inline' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com www.gstatic.com fonts.gstatic.com fonts.googleapis.com translate.google.com translate.googleapis.com player.vimeo.com; \
                        style-src               'self' 'unsafe-inline' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com www.gstatic.com fonts.gstatic.com fonts.googleapis.com translate.google.com translate.googleapis.com player.vimeo.com; \
                        style-src-elem          'self' 'unsafe-inline' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com www.gstatic.com fonts.gstatic.com fonts.googleapis.com translate.google.com translate.googleapis.com player.vimeo.com; \
                        style-src-attr          'self' 'unsafe-inline' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocoolie.com ted.com embed.ted.com embed-ssl.ted.com www.gstatic.com fonts.gstatic.com fonts.googleapis.com translate.google.com translate.googleapis.com player.vimeo.com; \
                        img-src                 'self' valheru.org www.valheru.org moredhel.org i.creativecommons.org licensebuttons.net fonts.gstatic.com translate.google.com translate.googleapis.com player.vimeo.com; \
                        font-src                'self' valheru.org www.valheru.org moredhel.org fonts.gstatic.com fonts.googleapis.com player.vimeo.com; \
                        connect-src             'self' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com stats.addtoany.com www.gstatic.com fonts.gstatic.com fonts.googleapis.com translate.google.com translate.googleapis.com player.vimeo.com; \
                        media-src               'self' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com player.vimeo.com; \
                        frame-src               'self' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com w.soundcloud.com static.addtoany.com www.dumpert.nl www.facebook.com m.facebook.com player.vimeo.com; \
                        frame-ancestors         'self' valheru.org www.valheru.org moredhel.org www.youtube.com youtube.com www.youtube-nocookie.com youtube-nocookie.com ted.com embed.ted.com embed-ssl.ted.com player.vimeo.com; \
                        form-action             'self' valheru.org www.valheru.org moredhel.org; \
                        base-uri                'self' valheru.org www.valheru.org moredhel.org; \
                        upgrade-insecure-requests; \
                        report-uri https://csp-reporter.valheru.org/report.php"
    Header always set Strict-Transport-Security "max-age=31536000;"
    Header always set Referrer-Policy "strict-origin"
    #Header always set Permissions-Policy "geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()"

    # Site settings
    AssignUserID valheru valheru
    DocumentRoot /home/valheru/webroot/valheru.org
    <Directory /home/valheru/webroot/valheru.org>
        Options -Indexes +FollowSymLinks +MultiViews +ExecCGI
        AllowOverride All
        Require all granted
    </Directory>

    CustomLog "|/usr/bin/rotatelogs -p /usr/local/bin/moveapachelogs /var/log/apache2/valheru.org-ssl-access.log.%Y%m%d%H 3600" combined
    #CustomLog "/var/log/apache2/valheru.org-ssl-access.log" combined
    ErrorLog "/var/log/apache2/valheru.org-ssl-error.log"
    LogLevel Warn

</VirtualHost>

I indeed see that the certificate is returned with a valid chain from my mailserver.
When generating the new certificate I did add OCSP stapling to the certificate and apache.
For this I added a line to the apache config

root@vps01:/tmp# cat /etc/apache2/conf-enabled/oscp.conf
SSLStaplingCache shmcb:/tmp/stapling_cache(128000)

And added SSLUseStapling on to all vhosts using this cert..

This one is an easy fix.

  1. Remove the line for SSLCertificateChainFile
  2. Change SSLCertificateFile from cert.pem to fullchain.pem

The SSLCertificateChainFile was deprecated many years ago. You only need ...File and ...KeyFile.

5 Likes

Now, let's look at this config file to check hekaya.nl (and probably others)
/etc/apache2/sites-enabled/010-valheru.org.conf

4 Likes

That has indeed fixed the problem. I will adjust all my vhosts accordingly.
Thank you very much for helping me find this error!

3 Likes

After adjusitng all vhosts like this my errors are gone, thank you so very much!

3 Likes

It isn't the identical error on the others. They were only sending the leaf and valheru was sending it twice. But, if the others were just using cert.pem instead of fullchain.pem that should fix it.

As for Must-Staple, have you enabled Apache's mod-md for that stapling? Do you know the possible issues involved with stapling? It can be very messy.

6 Likes

I have used stapling in the past for a video CDN I worked at as sysadmin. Never had any issues at that time. Never used it in my private setup though until yesterday so I am not aware of any issues. Can you enlighten me?

I have not enabled mod-md, I do have mod_ssl and mod_socache_shmbc, I will read up on mod-md.

Well, Apache stapling support is better directed at an Apache forum. There has been more discussion about that on this forum given Let's Encrypt's recent announcement: Intent to End OCSP Service - Let's Encrypt

Use this forum's Advanced Search for Must-Staple and look at recent threads

Perhaps also see these for further "food for thought"

https://httpd.apache.org/docs/current/mod/mod_md.html#mdstapleothers

https://blog.hboeck.de/archives/886-The-Problem-with-OCSP-Stapling-and-Must-Staple-and-why-Certificate-Revocation-is-still-broken.html

4 Likes

Thank you very much for this information, I will look in to it.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.