Help with Intermediate Cert Not Working

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: phd.nebulonenergy.org

I ran this command:

It produced this output:

My web server is (include version): Apache 2.4.58

The operating system my web server runs on is (include version): Ubuntu 24.04

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): YES

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.9.0

Some users that must go through their web proxy to reach https://phd.nebulonenergy.org/phd receive a SSL error (SEC_ERROR_UNKNOWN_ISSUER). Viewing the certificate on their browser, they do not see the intermediate certificate.

My apache configuration is simple:

<VirtualHost *:443>
  ServerName nebulonenergy.org
  ServerAlias phd.nebulonenergy.org
  Include /etc/letsencrypt/options-ssl-apache.conf
  SSLCertificateFile      /etc/letsencrypt/live/nebulonenergy.org/fullchain.pem
  SSLCertificateKeyFile   /etc/letsencrypt/live/nebulonenergy.org/privkey.pem
</VirtualHost>

I tried adding the following line
SSLCertificateChainFile /etc/letsencrypt/live/nebulonenergy.org/fullchain.pem
to no effect, and also
SSLCertificateChainFile /etc/letsencrypt/live/nebulonenergy.org/chain.pem
also to no effect.

Can someone point me in the right direction? Thanks.

2

apachectl -S
VirtualHost configuration:
*:80 nebulonenergy.org (/etc/apache2/sites-enabled/le-redirect-nebulonenergy.org.conf:1)
*:443 nebulonenergy.org (/etc/apache2/sites-enabled/phd.conf:20)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33 not_used
Group: name="www-data" id=33 not_used

3

Please show this whole file:

2 Likes
4

Looks like port 443 is routed to a "Mojolicious" server and not your Apache. HTTP (port 80) requests reach your intended Apache but not port 443. If this is a residential setup check your router's port forwarding or NAT configuration.

curl -ik https://phd.nebulonenergy.org
HTTP/2 302
server: Mojolicious (Perl)
location: /phd

curl -ik https://phd.nebulonenergy.org/phd
HTTP/2 302
server: Mojolicious (Perl)
location: /login

curl -ik https://phd.nebulonenergy.org/login
HTTP/2 200
content-type: text/html;charset=UTF-8
content-length: 2440
server: Mojolicious (Perl)
2 Likes
5 
<VirtualHost *:443>
    ServerName nebulonenergy.org
    ServerAlias phd.nebulonenergy.org
    DocumentRoot        /opt/phd/public/
    ##
    ## This sets up the Reverse Proxy to the app server
    ##
    ProxyPass           /   http://localhost:3001/ keepalive=On retry=0
    ProxyPassReverse    /   http://localhost:3001/
    LogLevel   Info
    Include                 /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile      /etc/letsencrypt/live/nebulonenergy.org/fullchain.pem
    SSLCertificateKeyFile   /etc/letsencrypt/live/nebulonenergy.org/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/nebulonenergy.org/chain.pem
</VirtualHost>
6

Sorry, forgot to mention that the Apache server is acting as a proxy to the application server, and I've included the entire phd.conf file which would have shown that.

7

Apache 2.4.58 does not need SSLCertificateChainFile when SSLCertificateFile's value is fullchain.pem.

I'm wondering if there are more VirtualHost configuration files giving issues. Please provide the output of sudo apachectl -t -D DUMP_VHOSTS. (Not sure if this would pick up more than apachectl -S but gotta try :man_shrugging:)

Also, let's check the Certbot certificate "store" with ls -l /etc/letsencrypt/live/nebulonenergy.org/ and ls -l /etc/letsencrypt/archive/nebulonenergy.org/

1 Like
8

Yeah, about the SSLCertificateChainFile. I've tried both with it and without it in the config, I'll remove that again.

$ sudo apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:80                   nebulonenergy.org (/etc/apache2/sites-enabled/le-redirect-nebulonenergy.org.conf:1)
*:443                  nebulonenergy.org (/etc/apache2/sites-enabled/phd.conf:20)

# ls -l /etc/letsencrypt/live/nebulonenergy.org
total 8
-rw-r--r-- 1 root root  692 Apr 23 23:20 README
lrwxrwxrwx 1 root root   41 Apr 23 23:20 cert.pem -> ../../archive/nebulonenergy.org/cert1.pem
lrwxrwxrwx 1 root root   42 Apr 23 23:20 chain.pem -> ../../archive/nebulonenergy.org/chain1.pem
lrwxrwxrwx 1 root root   46 Apr 23 23:20 fullchain.pem -> ../../archive/nebulonenergy.org/fullchain1.pem
-rw------- 1 root root 2883 Apr 23 23:27 nebulonenergy.org.pfx
lrwxrwxrwx 1 root root   44 Apr 23 23:20 privkey.pem -> ../../archive/nebulonenergy.org/privkey1.pem

# ls -l /etc/letsencrypt/archive/nebulonenergy.org/
total 16
-rw-r--r-- 1 root root 1371 Apr 23 23:20 cert1.pem
-rw-r--r-- 1 root root 1566 Apr 23 23:20 chain1.pem
-rw-r--r-- 1 root root 2937 Apr 23 23:20 fullchain1.pem
-rw------- 1 root root  241 Apr 23 23:20 privkey1.pem
9

Yeah, the info in your first post did not show that at all.

It looks very much like that other server is terminating the HTTPS connection and not Apache. Are you sure port 443 requests are being sent to Apache?

Check the Apache access_log to see if it is handling incoming requests.

Because the cert returned has no chain. All of the Apache changes you made should have shown something other than this. See: SSL Checker

It shows the Mojolicious server and the faulty chain. How is Mojolicious configured? Does it have any certificate config?

3 Likes
10

Is this directive anywhere in your apache config ?

SSLEngine on

It needs to be enabled in the server config or specific virtual hosts.

Also, do you have the other ssl directives set up? Protocols, ciphers, etc...

See:: Mozilla SSL Configuration Generator

3 Likes
11

Yeah, maybe the default Apache VHost is handling it

What does this show?

sudo grep -iR sslcertificate /etc/apache2/*

Although, if the default VHost was handling it then it must also be doing proxy as otherwise we wouldn't see the response header saying Mojolicious. We would also have to see Proxy Preserve Host which we don't see.

I really think the port routing is still the most likely reason.

2 Likes
12

The Include statement pulls in the following from /etc/letsencrypt/options-ssl-apache.conf
S

SLEngine on

# Intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder     off
SSLSessionTickets       off

SSLOptions +StrictRequire

# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common

which should turn on the SSL, set the ciphers, etc.

As for the backend app server: it listens on port 3001 and is only accepting connections from localhost.

I was missing the ProxyPreserveHost On configuration and have added that. Unfortunately no change.

# grep -iR sslcertificate /etc/apache2/*
/etc/apache2/sites-available/phd.conf:    SSLCertificateFile 	    /etc/letsencrypt/live/nebulonenergy.org/fullchain.pem
/etc/apache2/sites-available/phd.conf:    SSLCertificateKeyFile   /etc/letsencrypt/live/nebulonenergy.org/privkey.pem
/etc/apache2/sites-available/phd.conf:    # SSLCertificateChainFile /etc/letsencrypt/live/nebulonenergy.org/chain.pem
/etc/apache2/sites-available/default-ssl.conf:	#   SSLCertificateFile directive is needed.
/etc/apache2/sites-available/default-ssl.conf:	SSLCertificateFile      /etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/apache2/sites-available/default-ssl.conf:	SSLCertificateKeyFile   /etc/ssl/private/ssl-cert-snakeoil.key
/etc/apache2/sites-available/default-ssl.conf:	#   Point SSLCertificateChainFile at a file containing the
/etc/apache2/sites-available/default-ssl.conf:	#   the referenced file can be the same as SSLCertificateFile
/etc/apache2/sites-available/default-ssl.conf:	#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt
/etc/apache2/sites-enabled/phd.conf:    SSLCertificateFile 	    /etc/letsencrypt/live/nebulonenergy.org/fullchain.pem
/etc/apache2/sites-enabled/phd.conf:    SSLCertificateKeyFile   /etc/letsencrypt/live/nebulonenergy.org/privkey.pem
/etc/apache2/sites-enabled/phd.conf:    # SSLCertificateChainFile /etc/letsencrypt/live/nebulonenergy.org/chain.pem
13

NOTE: copy and paste error in options-ssl-apache.conf, it actually says SSLEngine on as you would want.

1 Like
14

If so, it must be on a gateway - otherwise Apache would fail trying to bind to the port.

The FQDN resolves to 4.154.230.56, which is allocated to Microsoft. The bare-ip serves a 404::

$ curl -k https://4.154.230.56/

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>Microsoft-Azure-Application-Gateway/v2</center>
</body>
</html>

$ curl -k -I https://4.154.230.56/

HTTP/2 404
server: Microsoft-Azure-Application-Gateway/v2
date: Wed, 18 Jun 2025 14:31:49 GMT
content-type: text/html
content-length: 179

@tbruner did you possibly configure something on the microsoft dashboard, or on a gateway server you control, to proxy onto the machine you are working with?

3 Likes
15

Agreed.

A port routing from 443 to, say, 3003 would avoid Apache port conflict.

But, yes, some external port / request routing issue not just duplicate servers on port 443

2 Likes
16

No, it wouldn't have.

Have you checked the Apache access_log to see if incoming requests are actually seen by that Apache VirtualHost?

Ideally add an access_log with a unique file name in that VHost

3 Likes
17

? Maybe ? Asked my infrastructure team for a VM. I'm new to this cloud stuff, and I was treating this system just like any old server. I'll start investigating that.

Here is what I see in the access log after clicking refresh on my browser:

nebulonenergy.org:80 10.165.101.135 - - [18/Jun/2025:14:41:45 +0000] "GET / HTTP/1.1" 301 425 "-" "-"
nebulonenergy.org:443 10.165.101.135 - - [18/Jun/2025:14:41:45 +0000] "GET /phd HTTP/1.1" 302 2995 "-" "-"

but I'm pretty sure that is not my IP, so maybe the the Azure Gateway is doing this?

18

"Cloud stuff"? "Infrastructure team"? "VM"?

You left the "hosting provider" question empty, probably because everything is "self hosted", but please be more forthcoming with potential relevant information.

Apparently we're not dealing with some attic hosted Raspberry Pi "self hosting", but a lot more elaborate. In that case, that would be relevant information, potentially crucial even.

The 10.0.0.0/8 IPv4 range is a private range, meaning not publicly routable. Thus, it seems your server is behind NAT. Anything can happen or be incorrectly configured on the device doing this NAT-ting.

That said, if it's logged in Apache, then the requests apparently reach it..

1 Like
19

I appreciate the help, I really do and I'm trying to be as forthcoming as I can. I'm not intentionally holding anything back, its just my poor understanding of azure and how I'm being hosted. I thought I was just on a VM that was accessible from the internet. Again I appreciate everyone's help here.

20

Yes, that is in a private/reserved netblock: 10.0.0.0/8. That's coming from some gateway in your system.

Assuming you have shell access on the server, try this:

curl -k --resolve phd.nebulonenergy.org:443:127.0.0.1 https://phd.nebulonenergy.org
curl -k -I --resolve phd.nebulonenergy.org:443:127.0.0.1 https://phd.nebulonenergy.org

That's just telling curl to resolve phd.nebulonenergy.org onto localhost, which should bypass the gateway. -k ignores certificate issues; -I shows headers.

If @MikeMcQ is right (and I agree with him), you should see the expected content and apache headers from this. That would pinpoint the routing issue to be before your server, and likely on the gateway.

3 Likes