I have a CentOS 6.9 machine with a handfull of domains on it served by Apache/2.2.15. I was having a spot of trouble getting Let’s Encrypt running, but I was finally able using certbot to get certificates for one domain set up (both the domain and the www. of the domain) and running properly, or at least I think so.
Again using certbot, I requested and received a certificate for a second domain on the server, which sends out the certificate for the first domain, which is not a good thing. The second domain I’m attempting is a test domain, so its failure isn’t deadly but I also want to install certs for other “real” domains on that server so I need to get this straightened out.
I suspect my problem might be the way my conf.d files are structured, but truthfully I dunno what the heck I did wrong. So any suggestions would be gratefully appreciated, and any requested information will be cheerfully supplied.
The biggest thing is to make sure you have each vhost configured in its own distinct conf file. Otherwise, Certbot will have issues parsing these domains and the relevant setup.
A grep on the directory says not; other than the .conf and -le-ssl.conf files for the two domain names that have certs from Let’s Encrypt. There are old files in a “workproduct” directory inside conf.d, but I’m pretty sure apache doesn’t recurse. Still, I’m moving that directory out and restarting just to make sure…yeah, same result, Untrusted Connection, loading second domain delivers certificate for the first domain.
Correct me if I’m wrong:
Apache 2.2.15 supports SNI.
The sites all worked prior to the addition of SSL/TLS.
and some sites now don’t work and the only change was adding SSL/TLS…
So, please show any relevant lines (which include any of the strings like: virtual, servername, & cert) from:
/etc/apache2/apache2.conf
/etc/apache2/conf-enabled/*
/etc/apache2/sites-enabled/*
where I suspect we can find the reason for the failures.
I believe so, yes, but I admit I don't really understand Server Name Indication, which is why I suspect the problem is in my config files. In the decades I've been working with computers, I have rarely gone wrong assuming I screwed something up...
Yep; and I suspect if I rem out the SSL redirect, both would work under http.
Only one thus far, although I assume if I added additional cert, the additional ones would fail as well, which would be a real issue. Currently the first name works, the second name doesn't, but only because there is a wrong cert error.
I mentioned I was running CentOS 6.9; there is no /etc/apache2 directory, it is instead /etc/httpd, and the two subdirectories you mention don't exist.
/etc/httpd/conf/httpd.conf - the file where in the good 'ol days all of the VirtualHost containers would have lived - is barren (for VirtualHosts, I mean) other than the remmed "example.com" ones. The VirtualHost files are in /etc/httpd/conf.d with one file per name ('cept the SSL names)...how much do you want to see? Should I include the first and second domain name files in their entirety? Neither are terribly large, each having two files (.conf and -le-ssl.conf). Ah, heck, this whole obfuscation is silly...
I’d like to see any and all lines that contain any the following words:
virtual
redirect
ssl
cert
servername
or just paste the entire vhost files that contain any of those words
There is nothing wrong with www.bobedwardsradio.com. The problem is solely with samples.lofcom.com. That BER is linking to an external http server's graphics files (and has some internal links that need to be updated, since it has been an http site for 9-10 years) is irrelevant to the main problem.
I should probably just delete the non-www cert, but nevermind that now. Again, this isn't the problem.
Wait. Again, you are dealing with the wrong issue. Please load http://samples.lofcom.com/ so you can see the redirect to https works and then the failure is apache delivering the www.bobedwardsradio.com certificate.
I suspect you are going to want to see both sets of config files, but please don't focus on BER serving a few non-https graphic files/links. It's samples.lofcom.com, and the likelihood that if I generate more certs they will do the same thing, that is the crisis.
Look again. It's https://bobedwardsradio.com/ that is failing - indeed, this is exactly the same problem, since click that link above and you'll get exactly the same error displayed on samples.lofcom.com. The server is serving one cert and one cert only; www.bobedwardsradio.com. So it doesn't matter what other cert I load into the server, it seems determined to serve that one certificate.
Config files follow; note the .conf files have been changed by certbot with redirect info, and the -le-ssl.conf created completely by certbot.
www.bobedwardsradio.com.conf:
<VirtualHost www.bobedwardsradio.com:80>
DocumentRoot "/home/admin/www.bobedwardsradio.com/web/"
ServerName www.bobedwardsradio.com
ServerAlias bobedwardsradio.com
<Directory /home/admin/www.bobedwardsradio.com/web/>
AllowOverride All
Options All
</Directory>
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.bobedwardsradio.com [OR]
RewriteCond %{SERVER_NAME} =bobedwardsradio.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [L,NE,R=permanent]
</VirtualHost>
www.bobedwardsradio.com-le-ssl.conf:
<IfModule mod_ssl.c>
<VirtualHost www.bobedwardsradio.com:443>
DocumentRoot "/home/admin/www.bobedwardsradio.com/web/"
ServerName www.bobedwardsradio.com
ServerAlias bobedwardsradio.com
<Directory /home/admin/www.bobedwardsradio.com/web/>
AllowOverride All
Options All
</Directory>
SSLCertificateFile /etc/letsencrypt/live/www.bobedwardsradio.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.bobedwardsradio.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/www.bobedwardsradio.com/chain.pem
</VirtualHost>
</IfModule>
samples.lofcom.com.conf:
<VirtualHost samples.lofcom.com:80>
ServerName samples.lofcom.com
DocumentRoot "/home/admin/samples.lofcom.com/web"
<Directory "/home/admin/samples.lofcom.com/web">
allow from all
AllowOverride All
Options +Indexes
</Directory>
RewriteEngine on
RewriteCond %{SERVER_NAME} =samples.lofcom.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [L,NE,R=permanent]
</VirtualHost>
samples.lofcom.com-le-ssl.conf:
<IfModule mod_ssl.c>
<VirtualHost samples.lofcom.com:443>
ServerName samples.lofcom.com
DocumentRoot "/home/admin/samples.lofcom.com/web"
<Directory "/home/admin/samples.lofcom.com/web">
allow from all
AllowOverride All
Options +Indexes
</Directory>
SSLCertificateFile /etc/letsencrypt/live/samples.lofcom.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/samples.lofcom.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/samples.lofcom.com/chain.pem
</VirtualHost>
</IfModule>
In looking at httpd.conf under "Section 3: Virtual Hosts," there is the following remmed section:
#
# Use name-based virtual hosting.
#
#NameVirtualHost *:80
#NameVirtualHost *:443
#
# NOTE: NameVirtualHost cannot be used without a port specifier
# (e.g. :80) if mod_ssl is being used, due to the nature of the
# SSL protocol.
#
...as well as a remmed-out example.com container, but nothing that would actually be processed. There are additional .conf files in the conf.d directory for the other sites served by that machine, but only the two above are supposed to serve as httpd.
Again, please note that file was created by certbot...
/etc/letsencrypt/options-ssl-apache.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.
SSLEngine on
# Intermediate configuration, tweak to your needs
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA2$
SSLHonorCipherOrder on
SSLOptions +StrictRequire
# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_$
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
#CustomLog /var/log/apache2/access.log vhost_combined
#LogLevel warn
#ErrorLog /var/log/apache2/error.log
# Always ensure Cookies have "Secure" set (JAH 2012/1)
#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4"
Try changing their domain names to real IP (removing required DNS lookup; as not all hostnames are not found in /etc/hosts) - like:
<VirtualHost 199.175.53.13:80>
<VirtualHost 199.175.53.13:443>
<VirtualHost 199.175.53.13:80>
<VirtualHost 199.175.53.13:443>
no matter what, one will always be the default when the system cant find a match (it will force one)
So if both vhost files fail yet one gets forced to match - appearing to be working properly.
Not as easy as you think; I tried that, and will try it again, but changing caused other errors.
First test: change the four files listed above only to contain *:(port). Created a bit of a disaster:
# /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: [Wed Oct 04 22:47:07 2017] [warn] _default_ VirtualHost overlap on port 443, the first has precedence
[Wed Oct 04 22:47:07 2017] [warn] _default_ VirtualHost overlap on port 80, the first has precedence
www.bobedwardsradio.com uses an invalid security certificate.
The certificate is not trusted because it is self-signed.
The certificate is only valid for vps.lofcom.com
Reverting back temporarily as that's production. And this does imply it's my configuration that is at fault, but unclear how to fix it.
/etc/hosts doesn't show any of the names of the web servers. Doesn't on any of the machines I control. Shows the default name of the machine and localhost. Since DNS is set up correctly, why would I need to enter the names in hosts? (Sincere question, since I never heard of doing that for apache virtual servers.)
[edited based on information from later post]
Since all names would always resolve to IP 199.175.53.13, why ask DNS?
If DNS ever fails, the system may fail.
VirtualHost is just the actual local IP and port binding.
Find the file that contains _default_ and you have the answer on why it "works" (sometimes).
I'm not thinking of either of those; but then that question was specifically why I would need to load the server's hosts file with information it can get from DNS.
Apparently I misunderstood. I thought you told me to replace the VirtualHost with the wildcard followed by the port, which I did to uncomfortable results. Did you want me to instead use:
<VirtualHost 127.0.0.1:80>
for all of the files (changing port in SSH, of course)?
Honest to god, I am completely confused. Are you suggesting apache doesn't know and cannot get the IP address of the server on which it resides? And even if that were the case, or the machine had multiple IP addresses attached, wouldn't:
VirtualHost *:80
...tell Apache it doesn't frelling matter what the IP address is the request comes in on?
Done, without changing anything else so that config files still read:
Stopping httpd: [ OK ]
Starting httpd: [Thu Oct 05 00:51:15 2017] [warn] VirtualHost samples.lofcom.com:80 overlaps with VirtualHost www.bobedwardsradio.com:80, the first has precedence, perhaps you need a NameVirtualHost directive
[ OK ]
...and neither of them are using certs other than the self-signed for default, vps.lofcom.com. The default is:
000_default.conf:
<VirtualHost 199.175.53.13:80>
ServerName vps.lofcom.com
DocumentRoot "/home/admin/bes.lofcom.com/web"
<Directory "/home/admin/bes.lofcom.com/web">
allow from all
Options +Indexes
</Directory>
</VirtualHost>
I should never see that unless something is very wrong. Which it is.
I am reverting hosts so that at least www.bobedwardsradio.com works overnight; I will get back to work on this in the morning. If I have to, I will delete all certificates, remove all VirtualHosts from the machine, and start over completely clean - getting to the point where I'm willing to remove apache itself, kill config files, and reinstall. But I need a pointer to the standard VHost config file, since what works fine for me in http across a number of servers when placed directly in the httpd.conf file, using server.name:80 instead of something_IP_related:80, clearly does not work at all when httpd is involved.
DNS will not normally match the local server IP - in this case it does.
But it's an unnecessary lookup when the IP should always be the local one.
Apache doesn't ask the server what is his IP it asks DNS; which may be completely external to this system and an unnecessary point of potential failure.
Basically yes. Unless your server has multiple IPs all request are going to one IP already.
It just says "I'm here and listening on this IP and port for this servername" - which must match a real local IP and port that it is LISTENing on.
You do need to read up on implementing SNI.
Here is an abbreviated snip from a single working Apache conf file where 41 domains exist on a single IP:
And yes they all also work flawlessly
1 is http only
7 are https only
and 33 are both - with simple redirects:
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
until we find every conf file in use (like the one with the _default_ ) you won't be able to fix the problem completely. Look for "NameVirtualHost" in the conf
You will only get part of it to work and only by accident.
Because it is not doing what you are telling it to do.
Apache is getting mixed signals.
Those vhost files are being ignored and only one was being used as an unmatched fallback
Which Apache will continue to do after the vhost names really don't match (but they already don't match - you just think they do).
Hardly magic; it's trivial for me to get the IP address that's bound to the network interface, so why should it be difficult for any other process on the server to get it as well? Fastest I know of is:
hostname I
...but there are others I don't know about, I'd bet.
]# hostname -I
127.0.0.1 199.175.53.13
Technically, yes, apache would cheerfully serve that name. It would never receive an inbound connection from the Internet for that VirtualHost, since DNS for google.com points elsewhere. But if I stuck an entry for google.com in my Windows machine's hosts file pointing to 199.175.53.13, set up a VirtualHost using ServerName google.com, and requested the site from my Windows machine, yes it would return whatever I set as DocumentRoot in the google.com VirtualHost. But a name has nothing to do with the bound IP address, so I digress.
There isn't one; that is, there's no NAT router ahead of the virtual machine (the provider has a filtering firewall and routers, of course). That's a static IP address they gave me (or rather it) when I leased the machine. This server is not running in my office or home, it's leased from a provider. dig -x 199.175.53.13, you get the reverse, vps.lofcom.com, my provider set up on their DNS when I leased the machine. This is set up no differently than the bare-metal servers I lease from Softlayer/IBM; a static IP address bound to the network interface.
If that is the case, then neither will work, as I proved the former does not. This is why I keep thinking I have something royally fouled up.
My fault for not being perfectly clear. This isn't some internal machine on a home network behind a bunch of NAT routers. It is a leased server with a static public IP address. There is no "internal" and "external" (not counting localhost), there is a single, real, honest-to-gd-dmned-g*d world-facing IP address.
I don't need to "find" anything, I have all of the apache config files sitting in two directories, conf and conf.d, and I already posted the default config file, which I named 000_default.conf so it would be included first by httpd.conf, and no, it doesn't declare default, and yes, it works just fine as the bloody http fallback - go to http://199.175.53.13/ and you'll see the webpage 000_default.conf points to. You really want me to post every blessed one of them here? Other than default, they all look pretty much the same other than the local DocumentRoot directories to which they point. But if you want to see them all, I'll post them all.
Right there, on the venet0:0 line, is the bound IP address we expect according to nslookup, and other than 127.0.0.1 there aren't any other IPv4 addresses anywhere in that result..
