Certificate chain fails after "certbot certonly ..."

Help
1

I'm getting the following complaint: "self signed certificate in certificate chain" for just one newly added domain for an LE certificate that continues to work fine for three other (related) domains.

I attempted to add 'covid.hoyo.zeetix.com' to the domains associated with the existing certificate. These are 'hoyo.zeetix.com', 'tms.hoyo.zeetix.com', and 'covid.tms.hoyo.zeetix.com'. The three existing domains continue to work fine with the existing certificate.

I used the "certbot certonly ..." command (see below) to add the single new domain. It apparently succeeded, because I see the following response to "certbot certificates" after the change:

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: hoyo.zeetix.com
    Serial Number: 3de392b7ebb82961e7afe0dafc11482aebd
    Key Type: ECDSA
    Domains: hoyo.zeetix.com covid.hoyo.zeetix.com covid.tms.hoyo.zeetix.com tms.hoyo.zeetix.com
    Expiry Date: 2023-09-26 22:56:48+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/hoyo.zeetix.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/hoyo.zeetix.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

When I attempt to use access the new domain, I get the aforementioned complaint. Here is the result of a failing curl command (from an ssh client on the same server):

$ curl -vk https://covid.hoyo.zeetix.com/
*   Trying 172.30.2.59...
* TCP_NODELAY set
* Connected to covid.hoyo.zeetix.com (172.30.2.59) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; O=Unspecified; CN=ip-172-30-0-50.ec2.internal; emailAddress=root@ip-172-30-0-50.ec2.internal
*  start date: Oct 27 23:32:12 2021 GMT
*  expire date: Nov  2 01:12:12 2022 GMT
*  issuer: C=US; O=Unspecified; OU=ca-9073428881288245187; CN=ip-172-30-0-50.ec2.internal; emailAddress=root@ip-172-30-0-50.ec2.internal
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
...

I've elided the result after the server failure.

When I compare this a successful response, I see that the following unexpected response:

* Server certificate:
*  subject: C=US; O=Unspecified; CN=ip-172-30-0-50.ec2.internal; emailAddress=root@ip-172-30-0-50.ec2.internal

On the same command using one of the old domains, this same entry is:

* Server certificate:
*  subject: CN=hoyo.zeetix.com

It appears to me that the 'certbot certonly' command somehow confused itself about the server certificate for the new domain (and JUST the new domain).

This is all well above my very limited expertise in such matters. I invite the help of this community in resolving this annoying issue.

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: hoyo.zeetix.com

I ran this command: (as root) certbot certonly --cert-name hoyo.zeetix.com -d hoyo.zeetix.com,covid.hoyo.zeetix.com,tms.hoyo.zeetix.com,covid.tms.hoyo.zeetix.com

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin (apache)
2: Runs an HTTP server locally which serves the necessary validation files under
the /.well-known/acme-challenge/ request path. Suitable if there is no HTTP
server already running. HTTP challenge only (wildcards not supported).
(standalone)
3: Saves the necessary validation files to a .well-known/acme-challenge/
directory within the nominated webroot path. A seperate HTTP server must be
running and serving files from the webroot path. HTTP challenge only (wildcards
not supported). (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 

** Invalid input **
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You are updating certificate hoyo.zeetix.com to include new domain(s):
+ covid.hoyo.zeetix.com

You are also removing previously included domain(s):
(None)

Did you intend to make this change?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(U)pdate certificate/(C)ancel: U
Renewing an existing certificate for hoyo.zeetix.com and 3 more domains

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/hoyo.zeetix.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/hoyo.zeetix.com/privkey.pem
This certificate expires on 2023-09-26.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

My web server is (include version): Apache/2.4.37 (rocky)

The operating system my web server runs on is (include version): "8.7 (Green Obsidian)

My hosting provider, if applicable, is: AWS EC2

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.6.0

2

Sounds like there was a coincidence in timing that points to certbot.
But it doesn't really make much sense to me that could happen from such a certbot command.
I'm leaning towards a misconfiguration in Apache.
Let's have a look at:
sudo apachectl -t -D DUMP_VHOSTS

3 Likes
3

Here is the command and response:

$ sudo apachectl -t -D DUMP_VHOSTS
Passing arguments to httpd using apachectl is no longer supported.
You can only start/stop/restart httpd using this script.
If you want to pass extra arguments to httpd, edit the
/etc/sysconfig/httpd config file.
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server covid.hoyo.zeetix.com (/etc/httpd/sites-enabled/covid-hoyo.conf:1)
         port 80 namevhost covid.hoyo.zeetix.com (/etc/httpd/sites-enabled/covid-hoyo.conf:1)
         port 80 namevhost covid.tms.hoyo.zeetix.com (/etc/httpd/sites-enabled/covid-tms-hoyo.conf:1)
         port 80 namevhost hoyo.zeetix.com (/etc/httpd/sites-enabled/hoyo.conf:1)
         port 80 namevhost tms.hoyo.zeetix.com (/etc/httpd/sites-enabled/tms-hoyo.conf:1)
         port 80 namevhost hoyo.zeetix.com (/etc/httpd/sites-available/hoyo.conf:1)
         port 80 namevhost tms.hoyo.zeetix.com (/etc/httpd/sites-available/tms-hoyo.conf:1)
         port 80 namevhost covid.tms.hoyo.zeetix.com (/etc/httpd/sites-available/covid-tms-hoyo.conf:1)
*:443                  is a NameVirtualHost
         default server zeetix.com (/etc/httpd/conf.d/ssl.conf:40)
         port 443 namevhost zeetix.com (/etc/httpd/conf.d/ssl.conf:40)
         port 443 namevhost hoyo.zeetix.com (/etc/httpd/sites-available/hoyo-le-ssl.conf:2)
         port 443 namevhost covid.tms.hoyo.zeetix.com (/etc/httpd/sites-available/covid-tms-hoyo-le-ssl.conf:2)
         port 443 namevhost tms.hoyo.zeetix.com (/etc/httpd/sites-available/tms-hoyo-le-ssl.conf:2)
4

Here is a list of issued certificates https://crt.sh/?q=covid.hoyo.zeetix.com (one).
Yet that is not the certificate being served up https://decoder.link/sslchecker/covid.hoyo.zeetix.com/443

1 Like
5

I appreciate your immediate attention and response!

I manually copied new .conf files in '.../sites-available' modified to reflect the new domain name, and added a symbolic link in '.../sites-enabled' analogous to the ones already there.

1 Like
6

Here is '.../sites-available/covid-hoyo-le-ssl.conf' that I added:

<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName covid.hoyo.zeetix.com
  DocumentRoot /var/www/hoyo/covid/html
  DirectoryIndex index.php index.htm index.html
  Alias /icons/ /var/www/icons/
  CustomLog "/var/log/httpd/covid.hoyo-access_log" combined
  ErrorLog  "/var/log/httpd/covid.hoyo-error_log"
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =covid.tms.hoyo.zeetix.com
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

SSLCertificateFile /etc/letsencrypt/live/hoyo.zeetix.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/hoyo.zeetix.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Here is the '.../sites-available/covid-hoyo.conf' that uses it:

<VirtualHost *:80>
  ServerName covid.hoyo.zeetix.com
  DocumentRoot /var/www/hoyo/covid/html
  DirectoryIndex index.php index.htm index.html
  Alias /icons/ /var/www/icons/
  CustomLog "/var/log/httpd/covid.hoyo-access_log" combined
  ErrorLog  "/var/log/httpd/covid.hoyo-error_log"
RewriteEngine on
RewriteCond %{SERVER_NAME} =covid.hoyo.zeetix.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

I've restarted 'httpd' multiple times (systemctl restart httpd) with no complaints.

I'm pretty sure these two new files follow the template of the existing (and working) ones.

7

Something is wrong in your main config file:
Please show:
sudo grep -i include /etc/apache2/apache2.conf

3 Likes
8

Heh ...

# grep -i include /etc/apache2/apache2.conf
grep: /etc/apache2/apache2.conf: No such file or directory

According to `find . -name "apache2.conf", the only references to 'apache2.conf' are in the vicinity of:

./var/lib/snapd/snap/certbot/3024/lib/python3.8/site-packages/certbot_apache/_internal/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/apache2.conf
9

Ok, your main config must be stored elsewhere...
find / -name apache2.conf
find / -name httpd.conf

3 Likes
10

I think you've found the issue -- I'm missing an INCLUDE in '/etc/httpd/conf/http.conf':

The following should be present and is not:

Include /etc/httpd/sites-available/conf-hoyo-le-ssl.conf

Sorry for jerking around the board, and I appreciate your help.

Once I edit and restart this, I'll close this topic when I confirm that the issue is solved.

11

That is not the way it should be done.
Nothing from /sites-available/ should ever be included directly.

Please show all the include lines.

4 Likes
12

These six lines are basically wrong:

4 Likes
13

Here is the last part of 'httpd.conf':

# Supplemental configuration
#
# Load config files in the "/etc/httpd/conf.d" directory, if any.
IncludeOptional conf.d/*.conf

# Handle subdomains
Include /etc/httpd/sites-enabled/*.conf
Include /etc/httpd/sites-available/hoyo.conf
Include /etc/httpd/sites-available/hoyo-le-ssl.conf

# covid app
Include /etc/httpd/sites-available/covid-hoyo.conf
Include /etc/httpd/sites-available/covid-hoyo-le-ssl.conf
Include /etc/httpd/sites-available/covid-tms-hoyo.conf
Include /etc/httpd/sites-available/covid-tms-hoyo-le-ssl.conf

Include /etc/httpd/sites-available/tms-hoyo.conf
Include /etc/httpd/sites-available/tms-hoyo-le-ssl.conf

Here is one of the original 'foo...conf'/'foo...-le-ssl.conf' pairs:

'hoyo.conf':

  ServerName hoyo.zeetix.com
  DocumentRoot /var/www/hoyo/html
  DirectoryIndex index.php index.htm index.html
  Alias /icons/ /var/www/icons/
  CustomLog "/var/log/httpd/hoyo-access_log" combined
  ErrorLog  "/var/log/httpd/hoyo-error_log"
  <Directory /var/www/hoyo/html/documentation>
    AcceptPathInfo on
    Options FollowSymlinks
    AllowOverride FileInfo
  </Directory>
RewriteEngine on
RewriteCond %{SERVER_NAME} =hoyo.zeetix.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

'hoyo-le-ssl.conf':

<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName hoyo.zeetix.com
  DocumentRoot /var/www/hoyo/html
  DirectoryIndex index.php index.htm index.html
  Alias /icons/ /var/www/icons/
  CustomLog "/var/log/httpd/hoyo-access_log" combined
  ErrorLog  "/var/log/httpd/hoyo-error_log"
  <Directory /var/www/hoyo/html/documentation>
    AcceptPathInfo on
    Options FollowSymlinks
    AllowOverride FileInfo
  </Directory>
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =hoyo.zeetix.com
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

SSLCertificateFile /etc/letsencrypt/live/hoyo.zeetix.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/hoyo.zeetix.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

I added the '<Directory...> pair myself.

14

I think I see origin of the disconnect, this is an artifact of the history of my use of LE.

I used the certbot apache command to generate the files in '.../sites-available' and '.../sites-enabled' on different systems ('byron.zeetix.com' and 'covid.zeetix.com') last year.

I checked my session logs (I keep all my SSH session logs), and here is the way I configured an analogous system ('byron.zeetix.com') in mid-February of 2022:

[root@byron ~]# certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: byron.zeetix.com
2: wiki.byron.zeetix.com
3: zeewiki.byron.zeetix.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): c
Please specify --domains, or --installer that will help in domain names autodiscovery, or --cert-name for an existing certificate name.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Those three domains were already configured using http.

I cloned (using AWS tools) the server that hosts 'byron.zeetix.com' to the new 'hoyo.zeetix.com'.

I had the probably mistaken idea that it would be easier to use the "certbot certonly ..." command on 'hoyo.zeetix.com', and that's probably where I got into trouble.

In any case, everything is working fine on 'hoyo.zeetix.com' now.

Since my company ("Zeetix, LLC") is likely to eventually offer many apps, each with its own subdomain, I use separate development servers so that the public-facing sites are not inadvertently damaged by issues during development of new apps.

I write too much. My takeaway is that using 'certbot apache' is probably the cleanest way to do all this in the future.

15

That is the only line you need there.
You should remove all the other includes with "/sites-available/".
As you should have noticed, those files are being loaded twice:
Example:

Then learn how to enable and disable config files within Aapche.

3 Likes
16

Got it, thanks.

I really appreciate your attention today.

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.