Occasional errors with my LetsEncrypt certs

On which web browser (and version) does this happen?

Well, now, thatā€™s interesting.

etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/sendy.amazinglybrilliant.com.au/fullchain.pem
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateKeyFile /etc/letsencrypt/live/sendy.amazinglybrilliant.com.au/privkey.pem
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/(snip).info/fullchain.pem
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateKeyFile /etc/letsencrypt/live/(snip).info/privkey.pem
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/next.radio/fullchain.pem
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateKeyFile /etc/letsencrypt/live/next.radio/privkey.pem
/etc/httpd/conf.d/ssl.conf:#   Point SSLCertificateFile at a PEM encoded certificate.  If
/etc/httpd/conf.d/ssl.conf:SSLCertificateFile /etc/pki/tls/certs/localhost.crt
/etc/httpd/conf.d/ssl.conf:SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
/etc/httpd/conf.d/ssl.conf:#   Point SSLCertificateChainFile at a file containing the
/etc/httpd/conf.d/ssl.conf:#   the referenced file can be the same as SSLCertificateFile
/etc/httpd/conf.d/ssl.conf:#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

So the one in ssl.conf might want to be looked at?

Yes, and no.
It probably needs to be there to catch all the unknowns (default).
I'm more interested in seeing the names used in each (and in the main host config).
[this should show HOW the confusion is occurring]
Please show:
grep -Eri 'servername|serveralias|virtualhost|SSLCertificateFile' /etc/apache2

[edit - maybe "confusion" is too strong a word]
[I think the fault is probably with the web client not the server]

Well, here we are. Apologies for the 'snipā€™s - I donā€™t want some of these addresses to be publicly visible.

/etc/httpd/conf/httpd-le-ssl.conf:<VirtualHost *:443>
/etc/httpd/conf/httpd-le-ssl.conf:Servername sendy.amazinglybrilliant.com.au
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/sendy.amazinglybrilliant.com.au/fullchain.pem
/etc/httpd/conf/httpd-le-ssl.conf:</VirtualHost>
/etc/httpd/conf/httpd-le-ssl.conf:<VirtualHost *:443>
/etc/httpd/conf/httpd-le-ssl.conf:ServerName (snip).info
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/(snip).info/fullchain.pem
/etc/httpd/conf/httpd-le-ssl.conf:</VirtualHost>
/etc/httpd/conf/httpd-le-ssl.conf:<VirtualHost *:443>
/etc/httpd/conf/httpd-le-ssl.conf:Servername next.radio
/etc/httpd/conf/httpd-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/next.radio/fullchain.pem
/etc/httpd/conf/httpd-le-ssl.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:# ports, instead of the default. See also the <VirtualHost>
/etc/httpd/conf/httpd.conf:# <VirtualHost> definition.  These values also provide defaults for
/etc/httpd/conf/httpd.conf:# any <VirtualHost> containers you may define later in the file.
/etc/httpd/conf/httpd.conf:# All of these directives may appear inside <VirtualHost> containers,
/etc/httpd/conf/httpd.conf:# ServerName gives the name and port that the server uses to identify itself.
/etc/httpd/conf/httpd.conf:#ServerName www.example.com:80
/etc/httpd/conf/httpd.conf:# If you do not specify an ErrorLog directive within a <VirtualHost>
/etc/httpd/conf/httpd.conf:# logged here.  If you *do* define an error logfile for a <VirtualHost>
/etc/httpd/conf/httpd.conf:    # If you do not define any access logfiles within a <VirtualHost>
/etc/httpd/conf/httpd.conf:    # define per-<VirtualHost> access logfiles, transactions will be
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername (snip2).info
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername (snip3).podnews.net
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername (snip4).james.crid.land
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername sendy.amazinglybrilliant.com.au
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername next.radio
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername (snip5).info
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:8080>
/etc/httpd/conf/httpd.conf:Servername (snip6).info
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:80>
/etc/httpd/conf/httpd.conf:Servername (snip7).info
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:8080>
/etc/httpd/conf/httpd.conf:Servername (snip8).info
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf/httpd.conf:<VirtualHost *:8080>
/etc/httpd/conf/httpd.conf:ServerName (snip9).mediauk.com
/etc/httpd/conf/httpd.conf:</VirtualHost>
/etc/httpd/conf.d/ssl.conf:<VirtualHost _default_:443>
/etc/httpd/conf.d/ssl.conf:#ServerName www.example.com:443
/etc/httpd/conf.d/ssl.conf:#   Point SSLCertificateFile at a PEM encoded certificate.  If
/etc/httpd/conf.d/ssl.conf:SSLCertificateFile /etc/pki/tls/certs/localhost.crt
/etc/httpd/conf.d/ssl.conf:#   the referenced file can be the same as SSLCertificateFile
/etc/httpd/conf.d/ssl.conf:</VirtualHost>

Ok so there doesnā€™t seem to be a systemwide ServerName.
[so that is not part of the confusion]

The file /etc/httpd/conf.s/ssl.conf has no ServerName but uses the _default_ which will match all unknowns.

Iā€™m pretty certain the problem is with the (old) web clients.
[that donā€™t support SNI]

That said, you have some options (if they are unable to update their web browsers).
If you know for a fact which domain(s) are being used by such clients: You can ā€œforceā€ that config on them. Such clients basically connect to the IP and expect content from that name (but they donā€™t pass the name to the server - so the server has a very low chance of providing the correct content/site). The web server will merely find the closest match (or the first config that matches the IP and port).
So you can remove the /etc/httpd/conf.d/ssl.conf file and move the vhost config block up (to first place) in the file /etc/httpd/conf/httpd-le-ssl.conf

Or (like in a cases where there are several sites that need to be first - but you only have one IP) you could attempt to ā€œcombineā€ them for such clients. That would entail obtaining a single cert with those names on it. Creating a ā€œdefaultā€ or ā€œfirstā€ config that points then to a ā€œmenuā€ page and allows them to then click on the actual site they need. But the links arenā€™t to the names (that fails for them); they are to folders in this vhost. Each of which is just an separate alias to the actual folder on the drive.
So that (basically)
https://SYSTEM.IP/site1 connects to content of the DocumentRoot for Site1
https://SYSTEM.IP/site2 connects to content of the DocumentRoot for Site2
ā€¦

1 Like

Nope. You may be thinking of Cloudflare, their free service is SNI only.

(It's also possible that you're not using a dedicated IP address but your site happens to be the only or first one using that IP address right now, which would have the same effect though it might not be permanent in that case).

podnews.net is an SNI-only Cloudfront distribution. This can be tested by sending a ClientHello with no SNI extension.

Well that's precisely what will happen when a client connects without SNI right?

I think, if that one is removed/disabled then the next one will take over as the default.

Edit now that I am fully awake I see that you have already explained this :slight_smile:

1 Like

Interesting, it worked without SNI when I checked yesterday. Must have just been luck.

Back again. The person with the ESET software sends me this ultra-confusing response:

Hi James, these links are still not working and it isnā€™t ESET, Iā€™ve just forwarded this email to my GMAIL account and opened it on a standard Chromebook which has no additional security software installed and Iā€™m getting exactly the same message:

sendy.amazinglybrilliant.com.au sent an invalid response
Try running connectivity diagnostics
ERR_SSL_PROTOCOL_ERROR

I wonder if something is changing the links before I even get them? Hereā€™s what I get for the link to the article about the BBC Sounds App
Has Bill Simmonsā€™ The Ringer figured out the model for podcast success? | Nieman Journalism Lab

I click on this link, and it bounces me through to the story just fine.

I've gone back and asked him to tell me a bit more about his internet connection, but for now, I'm more confused than ever!

Hereā€™s the (quotable-printable encoded) email that he was sent. Doesnā€™t look like thereā€™s anything wrong hereā€¦

23%20pm

So Iā€™m not sure if this is related but weā€™ve recently diagnosed that some versions of CentOS have a problem where it matters whether the domain name is before or after ssl alphabetically (!!):

Could you check whether this could be related to your situation? It looks like youā€™re using CentOS, at least.

(If so, you could also be in a position to test my proposed solution to this Certbot bug!)

Iā€™m usingā€¦

uname -a
Linux ip-172-30-0-169 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

ā€¦which is Amazon Linux. (Which, I think, is CentOS under another name, but I might be wrong).

Can you see what the main Include looks like in your Apache configuration and what the ssl.conf file looks like?

Sure! In order, here are the Include statements in my httpd.conf file.

Include conf.modules.d/*.conf
IncludeOptional conf.d/*.conf
Include /etc/httpd/conf/httpd-le-ssl.conf

Inside the conf.modules.d directory, 00-ssl.conf has one line in it:

LoadModule ssl_module modules/mod_ssl.so

The conf.d/ssl.conf looks like:

#
# When we also provide SSL we have to listen to the
# standard HTTPS port in addition.
#
Listen 443 https

##
##  SSL Global Context
##
##  All SSL configuration in this context applies both to
##  the main server and all SSL-enabled virtual hosts.
##

#   Pass Phrase Dialog:
#   Configure the pass phrase gathering process.
#   The filtering dialog program (`builtin' is a internal
#   terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog

#   Inter-Process Session Cache:
#   Configure the SSL Session Cache: First the mechanism
#   to use and second the expiring timeout (in seconds).
SSLSessionCache         shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout  300

#   Pseudo Random Number Generator (PRNG):
#   Configure one or more sources to seed the PRNG of the
#   SSL library. The seed data should be of good random quality.
#   WARNING! On some platforms /dev/random blocks if not enough entropy
#   is available. This means you then cannot use the /dev/random device
#   because it would lead to very long connection times (as long as
#   it requires to make more entropy available). But usually those
#   platforms additionally provide a /dev/urandom device which doesn't
#   block. So, if available, use this one instead. Read the mod_ssl User
#   Manual for more details.
SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
#SSLRandomSeed startup file:/dev/random  512
#SSLRandomSeed connect file:/dev/random  512
#SSLRandomSeed connect file:/dev/urandom 512

#
# Use "SSLCryptoDevice" to enable any supported hardware
# accelerators. Use "openssl engine -v" to list supported
# engine names.  NOTE: If you enable an accelerator and the
# server does not start, consult the error logs and ensure
# your accelerator is functioning properly.
#
SSLCryptoDevice builtin
#SSLCryptoDevice ubsec

##
## SSL Virtual Host Context
##

<VirtualHost _default_:443>

# General setup for the virtual host, inherited from global configuration
#DocumentRoot "/var/www/html"
#ServerName www.example.com:443

# Use separate log files for the SSL virtual host; note that LogLevel
# is not inherited from httpd.conf.
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   List the protocol versions which clients are allowed to connect with.
#   Disable SSLv3 by default (cf. RFC 7525 3.1.1).  TLSv1 (1.0) should be
#   disabled as quickly as practical.  By the end of 2016, only the TLSv1.2
#   protocol or later should remain in use.
SSLProtocol all -SSLv3
SSLProxyProtocol all -SSLv3

#   User agents such as web browsers are not configured for the user's
#   own preference of either security or performance, therefore this
#   must be the prerogative of the web server administrator who manages
#   cpu load versus confidentiality, so enforce the server's cipher order.
SSLHonorCipherOrder on

#   SSL Cipher Suite:
# List the ciphers that the client is permitted to negotiate.
# See the mod_ssl documentation for a complete list.

# By leaving this directive commented out, the system-wide OpenSSL
# default is used.  See update-crypto-policies(8) for more details.
#SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
#SSLProxyCipherSuite HIGH:MEDIUM:!aNULL:!MD5

#   Point SSLCertificateFile at a PEM encoded certificate.  If
#   the certificate is encrypted, then you will be prompted for a
#   pass phrase.  Note that restarting httpd will prompt again.  Keep
#   in mind that if you have both an RSA and a DSA certificate you
#   can configure both in parallel (to also allow the use of DSA
#   ciphers, etc.)
#   Some ECC cipher suites (http://www.ietf.org/rfc/rfc4492.txt)
#   require an ECC certificate which can also be configured in
#   parallel.
SSLCertificateFile /etc/pki/tls/certs/localhost.crt

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
#   ECC keys, when in use, can also be configured in parallel
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

#   Server Certificate Chain:
#   Point SSLCertificateChainFile at a file containing the
#   concatenation of PEM encoded CA certificates which form the
#   certificate chain for the server certificate. Alternatively
#   the referenced file can be the same as SSLCertificateFile
#   when the CA certificates are directly appended to the server
#   certificate for convenience.
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt

#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth  10

#   Access Control:
#   With SSLRequire you can do per-directory access control based
#   on arbitrary complex boolean expressions containing server
#   variable checks and other lookup directives.  The syntax is a
#   mixture between C and Perl.  See the mod_ssl documentation
#   for more details.
#<Location />
#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
#</Location>

#   SSL Engine Options:
#   Set various options for the SSL engine.
#   o FakeBasicAuth:
#     Translate the client X.509 into a Basic Authorisation.  This means that
#     the standard Auth/DBMAuth methods can be used for access control.  The
#     user name is the `one line' version of the client's X.509 certificate.
#     Note that no password is obtained from the user. Every entry in the user
#     file needs this password: `xxj31ZMTZzkVA'.
#   o ExportCertData:
#     This exports two additional environment variables: SSL_CLIENT_CERT and
#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
#     server (always existing) and the client (only existing when client
#     authentication is used). This can be used to import the certificates
#     into CGI scripts.
#   o StdEnvVars:
#     This exports the standard SSL/TLS related `SSL_*' environment variables.
#     Per default this exportation is switched off for performance reasons,
#     because the extraction step is an expensive operation and is usually
#     useless for serving static content. So one usually enables the
#     exportation for CGI and SSI requests only.
#   o StrictRequire:
#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
#     under a "Satisfy any" situation, i.e. when it applies access is denied
#     and no other module can change it.
#   o OptRenegotiate:
#     This enables optimized SSL connection renegotiation handling when SSL
#     directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars
</Directory>

#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait for
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is sent or allowed to be received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
#   o ssl-accurate-shutdown:
#     This forces an accurate shutdown when the connection is closed, i.e. a
#     SSL close notify alert is sent and mod_ssl waits for the close notify
#     alert of the client. This is 100% SSL/TLS standard compliant, but in
#     practice often causes hanging connections with brain-dead browsers. Use
#     this only for browsers where you know that their SSL implementation
#     works correctly.
#   Notice: Most problems of broken clients are also related to the HTTP
#   keep-alive facility, so you usually additionally want to disable
#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
#   "force-response-1.0" for this.
BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

#   Per-Server Logging:
#   The home of a custom SSL log file. Use this when you want a
#   compact non-error SSL logfile on a virtual host basis.
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

@joohoi, do you think this is the same problem as with CentOS 6, or a different problem?

Well, my apologies to you all for wasting your time.

It seems that Trend Micro have blocked sendy.amazinglybrilliant.com.au as a ā€œspamā€ website, on this chapā€™s router. So, what youā€™re seeing in the comments above is presumably an attempt to display some kind of ā€œThis website has been blockedā€ text, which of course doesnā€™t appear because itā€™s https only.

For the benefit of others who might stumble upon this thread in later years - here is the Trend Micro website to change the classification of a website. Anyone can do it. (Should you want to do that on my behalf, please feel free - Iā€™ve suggested ā€œNews/Mediaā€ as the new content classification.)

For the record, the privacy statement is over here and discusses double opt-in measures for the two mailing lists that this service runs.

How frustrating - but at least Iā€™ve got to the bottom of it. Apologies once more for wasting everyoneā€™s time.

2 Likes

Terrible.

But good to know - thanks!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.