Err Connection Reset: Accessing my HTTPS Website from Corporate Proxy


#1

Introduction

My website is inaccessible from a particular workplace proxy upon activating a HTTPS encryption. Switch back to a HTTP connection and it works. I have also confirmed that the website is working from an outside internet connection and Qualys SSL Labs issued an A+ Rating.

Server Configuration

  • Ubuntu 16.04 with Apache 2.4.18
  • SSL Certificates issued from Let’s Encrypt (letsencrypt.org)

Ruled Out

  • Certificate Provider: Managed to access other sites encrypted with Let’s Encrypt certificates.
  • Keyword Block: This would mean that my HTTP site would not be accessible as well but that is not the case.

Examples

Error Logs

I have managed to capture the error logs when a client from the corporate proxy tries to access the website.

[Fri Sep 16 10:45:13.850364 2016] [ssl:info] [pid 29221] [client ip.xxx.xxx.xxx] AH01964: Connection to child 3 established (server website.com:443)
[Fri Sep 16 10:45:13.850692 2016] [ssl:trace2] [pid 29221] ssl_engine_rand.c(126): Seeding PRNG with 656 bytes of entropy
[Fri Sep 16 10:45:13.850800 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1970): [client ip.xxx.xxx.xxx] OpenSSL: Handshake: start
[Fri Sep 16 10:45:13.850828 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: before/accept initialization
[Fri Sep 16 10:45:13.850907 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: read 11/11 bytes from BIO#562b352aea90 [mem: 562b352bc180] (BIO dump follows)
[Fri Sep 16 10:45:13.850977 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: read 175/175 bytes from BIO#562b352aea90 [mem: 562b352bc18e] (BIO dump follows)
[Fri Sep 16 10:45:13.851043 2016] [ssl:debug] [pid 29221] ssl_engine_kernel.c(2096): [client ip.xxx.xxx.xxx] AH02043: SSL virtual host for servername website.com found
[Fri Sep 16 10:45:13.851059 2016] [core:debug] [pid 29221] protocol.c(1891): [client ip.xxx.xxx.xxx] select protocol from , choices=h2,spdy/3.1,http/1.1 for server website.com
[Fri Sep 16 10:45:13.851074 2016] [ssl:debug] [pid 29221] ssl_engine_kernel.c(2096): [client ip.xxx.xxx.xxx] AH02043: SSL virtual host for servername website.com found
[Fri Sep 16 10:45:13.851129 2016] [ssl:debug] [pid 29221] ssl_util_stapling.c(754): AH01951: stapling_cb: OCSP Stapling callback called
[Fri Sep 16 10:45:13.851167 2016] [ssl:debug] [pid 29221] ssl_util_stapling.c(762): AH01952: stapling_cb: retrieved cached certificate data
[Fri Sep 16 10:45:13.851188 2016] [socache_shmcb:debug] [pid 29221] mod_socache_shmcb.c(528): AH00835: socache_shmcb_retrieve (0xef -> subcache 3)
[Fri Sep 16 10:45:13.851199 2016] [socache_shmcb:debug] [pid 29221] mod_socache_shmcb.c(880): AH00849: match at idx=0, data=0
[Fri Sep 16 10:45:13.851207 2016] [socache_shmcb:debug] [pid 29221] mod_socache_shmcb.c(538): AH00836: leaving socache_shmcb_retrieve successfully
[Fri Sep 16 10:45:13.851235 2016] [ssl:debug] [pid 29221] ssl_util_stapling.c(314): AH01933: stapling_get_cached_response: cache hit
[Fri Sep 16 10:45:13.851244 2016] [ssl:debug] [pid 29221] ssl_util_stapling.c(697): AH01953: stapling_cb: retrieved cached response
[Fri Sep 16 10:45:13.851330 2016] [ssl:debug] [pid 29221] ssl_util_stapling.c(813): AH01956: stapling_cb: setting response
[Fri Sep 16 10:45:13.851348 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.851366 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: write 117/117 bytes to BIO#562b352ba920 [mem: 562b352c42d3] (BIO dump follows)
[Fri Sep 16 10:45:13.851422 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.851452 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: write 2472/2472 bytes to BIO#562b352ba920 [mem: 562b352bc183] (BIO dump follows)
[Fri Sep 16 10:45:13.851487 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.851497 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: write 540/540 bytes to BIO#562b352ba920 [mem: 562b352bc183] (BIO dump follows)
[Fri Sep 16 10:45:13.851512 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.855654 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: write 338/338 bytes to BIO#562b352ba920 [mem: 562b352bc183] (BIO dump follows)
[Fri Sep 16 10:45:13.855690 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.855703 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: write 9/9 bytes to BIO#562b352ba920 [mem: 562b352bc183] (BIO dump follows)
[Fri Sep 16 10:45:13.855713 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.855730 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2056): [client ip.xxx.xxx.xxx] OpenSSL: write 3476/3476 bytes to BIO#562b352b9870 [mem: 562b352c9340] (BIO dump follows)
[Fri Sep 16 10:45:13.855741 2016] [core:trace6] [pid 29221] core_filters.c(525): [client ip.xxx.xxx.xxx] core_output_filter: flushing because of FLUSH bucket
[Fri Sep 16 10:45:13.855844 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:13.855861 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(1979): [client ip.xxx.xxx.xxx] OpenSSL: Loop: unknown state
[Fri Sep 16 10:45:14.215543 2016] [ssl:trace4] [pid 29221] ssl_engine_io.c(2065): [client ip.xxx.xxx.xxx] OpenSSL: I/O error, 5 bytes expected to read on BIO#562b352aea90 [mem: 562b352bc183]
[Fri Sep 16 10:45:14.215665 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(2008): [client ip.xxx.xxx.xxx] OpenSSL: Exit: error in unknown state
[Fri Sep 16 10:45:14.215688 2016] [ssl:trace3] [pid 29221] ssl_engine_kernel.c(2008): [client ip.xxx.xxx.xxx] OpenSSL: Exit: error in unknown state
[Fri Sep 16 10:45:14.215723 2016] [ssl:debug] [pid 29221] ssl_engine_io.c(1227): (70014)End of file found: [client ip.xxx.xxx.xxx] AH02007: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]
[Fri Sep 16 10:45:14.215763 2016] [ssl:info] [pid 29221] [client ip.xxx.xxx.xxx] AH01998: Connection closed to child 3 with abortive shutdown (server website.com:443)

Would gladly appreciate if anyone could shed some light on why some HTTPS websites do not work through the corporate proxy.


#2

A+ on SSL Labs indicates that your server uses fairly modern cipher suites. It’s possible that your proxy is unable to use any of these, similar to how older web browser (think: IE 6) would be unable to visit such a site.

My recommendation would be try the cipher suites recommended by Mozilla’s SSL Configuration Generator in the “Intermediate” or “Old” setting. To do that, you’d replace the SSLProtocol and SSLCipherSuite directives in your apache configuration with the values provided by that site.

If things work with the intermediate settings, you should be okay with those. These ciphers aren’t the best out there, but plenty of big sites think they’re good enough. If things only work with the old settings, I’d definitely look into updating or replacing that proxy rather than changing your server configuration, as those ciphers and protocol are considered fairly insecure nowadays.


#3

@pfg, that’s a good suggestion.

In 2012 Jeff Jarmoc pointed out that using proxies this way can result in diminished security because the proxies’ TLS clients are weaker against some security threats than the browsers’ would be. (One reason can simply be that the proxy software probably isn’t updated nearly as frequently as browser software would be.)


#4

I’ve taken your recommendation and tested it with the following settings on Mozzilla’s SSL Configuration Generator.

Unfortunately, even with the “Old” setting, the site is still not working for those users behind the proxy. It still shows an A+ on SSL Labs though. I’ll be leaving the server configured with the cipher suite recommended for any further testing if needed for the next few days.

Perhaps there could be something else I missed out? Twitter is also graded an A+ and the site works for the users.

Error Messages

Internet Explorer - "The proxy server isn’t responding"
Google Chrome - "This site can’t be reached. The connection was reset. [ERR_CONNECTION_RESET]"
Firefox - "Secure Connection Failed"

Update

I realised that my OCSP is not accessible.
OCSP: http://ocsp.int-x3.letsencrypt.org/

Twitter has their CRL & OCSP that is accessible.
CRL: http://crl3.digicert.com/sha2-ev-server-g1.crl
OCSP: http://ocsp.digicert.com


#5

This indicates that the change did not go into effect. Was the domain in question https://www.circles.life/? I just checked and noticed that this domain is hosted behind CloudFlare. In that case, your browser does not connect directly to your server, but rather to CloudFlare. CloudFlare has a separate certificate for your domain (not from Let’s Encrypt), and you can’t really change the TLS configuration since that’s up to CloudFlare. At least for this domain, Let’s Encrypt is not really part of the equation.

What kind of error are you getting? Could you run traceroute ocsp.int-x3.letsencrypt.org if you can’t connect at all?


#6

I can confirmed the changes were applied as changing the cipher suite to ALL:+LOW caused the grading to be capped at B on SSL Labs. I’ve attached my configuration file for your reference.

<IfModule mod_ssl.c>
<VirtualHost *:443>

SSLCertificateFile /.../fullchain.pem
SSLCertificateKeyFile /.../privkey.pem

ServerName techlingo.sg

Header always set Strict-Transport-Security "max-age=15768000"

</VirtualHost>

# intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv3
SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
#SSLCipherSuite          ALL:+LOW
SSLHonorCipherOrder     on
SSLCompression          off
SSLSessionTickets       off

# OCSP Stapling, only in httpd 2.3.3 and later
SSLUseStapling          off
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
SSLStaplingCache        shmcb:/.../ocsp(128000)

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
</IfModule>

No it is https://techlingo.sg

There were no error messages and the browser just shows a blank page. Other OSCP sites will download a file. Anyway, OCSP Stapling is disabled and I believe it shouldn’t be the cause of this issue.

Apparently the tracert command in cmd will not work behind the proxy as they might have blocked some of the protocols. Running it outside the proxy shows the destination at 101.100.190.154 which I believe is not the actual destination.


#7

The empty page you’re getting is fine. A real OCSP query would send other parameters (i.e. not simply do a GET /).

Let’s try the exact same cipher suites used by helloworld.letsencrypt.org:

SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA

If that doesn’t change anything, we know it’s likely not protocol/cipher-related.

There are a few other differences between those servers, most of which I don’t think would have any effect on a proxy, but who knows what these things care about. :smile: Here’s a list:

  • helloworld.letsencrypt.org supports session resumption.
  • helloworld.letsencrypt.org supports OCSP stapling. If the proxy indeed has problems talking to Let’s Encrypt’s OCSP but supports stapling, that could be an explanation. I think this should be enough to enable stapling:
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
SSLStaplingCache        shmcb:/var/run/ocsp(128000)```
 - `helloworld.letsencrypt.org` supports NPN, your server supports ALPN. This is related to the `openssl` version you're using, NPN is supported by older versions of `openssl` while newer ones support ALPN. If you want to rule out that this is the cause, you'd have to spin up a new web server with `openssl` < `1.0.2`, e.g. Ubuntu 14.04. Wouldn't bother with this until everything else is ruled out.

#8

Well that’s unfortunate… It still does not work behind the proxy :sob:

I have also implemented OCSP stapling on our site and we are still facing the issue.

That leaves us with session resumption and NPN support?


#9

Very odd. :confused:

I don’t think session resumption and NPN are likely candidates, as this should be a fairly common configuration, so odds are many sites would break if your proxy had problems with that. Then again, this is the last TLS-related difference between your site and helloworld.letsencrypt.org I can think of, so who knows, maybe it’s a weird combination of things and one of these two is actually the culprit.


#10

Very odd indeed.

I totally agree. But then again, I hear there are quite a number of sites that are not accessible with no observable pattern.

Would it be relatively easy to implement session resumption or NPN to see if it might be the cause?


#11

Update:

I’ve found another website supporting session resumption and NPN that is not accessible behind the proxy.


#12

Update 2:

A few of my colleagues realised it may be due to the root certificate not being trusted by the proxy server. Namely DST Root CA X3 and AddTrust External CA Root. The root certificate is the next thing that is similar across all sites that doesn’t work behind the proxy which we’ve previously mentioned.

Why https://helloworld.letsencrypt.org still works may be due to a 2nd certificate that is shown in SSL Labs from IdenTrust Commercial Root CA 1. Guess the next thing to try changing our Certification Authority and I’ll keep you updated.


#13

The second certificate shown by SSL Labs should only be relevant if your proxy doesn’t support SNI - otherwise, it would only see the certificate chaining up to the DST root. If it actually doesn’t support SNI and gets the certificate chaining up to “IdenTrust Commercial Root CA 1”, it should still fail as that certificate doesn’t include the helloworld.letsencrypt.org domain (only letsencrypt.org and www.letsencrypt.org). Then again, based on the things these proxies get wrong sometimes, it wouldn’t surprise me if they skipped the hostname check. :slight_frown:


#14

You are right. Even after trying with a different Certification Authority, the website is still not accessible.

I think we would have to look at other areas since we’ve ruled out this list. I’ll continue researching :slight_smile: By the way, the domain in question has been moved to https://techlingo.sg


#15

Is there a possibility that the interactive installation that I used in the beginning could have modified some of the SSL configuration?

I was intending to start another server from scratch but would like to eliminate that possibility first.


#16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.