I’ve been attempting to troubleshoot a particularly elusive problem for some time now, it’s had me stumped for months, I’m really grasping at straws here, but hopefully someone has seen something like this before, so bear with me while I lay out the scenario…
My company has a particular Website (secured by SSL) that sits on a number of identical (physical) web servers, that are connected directly to a Cisco Content Services Switch (11501), which is in-turn directly connected to a Cisco ASA Firewall, which then connects out to the internet. ( [Internet] > [ASA Firewall] > [Content Switch] > [Web Server Cluster] )
The Content switch serves two purposes, to terminate SSL handshakes from clients, so that clients authenticate with the switch directly, and to load balance incoming connection between the web servers in the cluster. We have 3 setups exactly the same as this, (A Production network, a testing and a disaster recovery network), and all use identical software and hardware.
The website and SSL handshake functions perfectly for all of our networks, EXCEPT when using Android devices (stock browser) and only to our production network. In this particular scenario, the android device fails to authenticate SSL with the Switch, and throws a “A secure connection could not be established” error message. (Note: this is not an actual SSL certificate error, Strange right!).
When I run many of the online SSL certificate checkers on the website (Such as this) it says that there is NO certificate chain present. So naturally, I came to the conclusion that the content switch was not storing or serving the SSL certs correctly. So after screwing around with new certs and rewriting the config for the switch to no avail, I replaced the entire switch with the one from the test network. Same problem. I then took the production switch, and put it in the test network, and it worked fine.
So basically it can’t be the switch.
So that leaves the Internet connection, firewall and/or servers. But it can’t be the servers, because SSL terminates at the switch. It also can’t be the internet connection, because our test network and production network both connect to the same ISP node, and carrier network. Which only leaves the firewall! So my ultimate question is this. How is it possible, that this Cisco ASA 5520 could be modifying encapsulated data in such a way, that would break HTTPS connections, but only on when using the stock browser on android devices. And/or cause some SSL cert checkers to think the certificate chain doesn’t exist?
Any thoughts?