SSL doesn't resolve only in Uzbekistan


#1

We’ve been encountering a strange anomaly lately. And we need your help urgently.
There’s a standart Vesta CP installed on our Debian powered VDS and Let’s Encrypt SSL is properly configured. Our domain https://old.grantlar.uz (whose DNS is also properly and completely configured) is functioning normally when opened anywhere around the globe, except for Uzbekistan.
Whenever we try to access https://old.grantlar.uz from inside Uzbekistan Chrome (and other majorly used browsers) show ERR_SSL_PROTOCOL_ERROR error. But if you turn on VPN which is based in USA or Europe, the website opens properly and browser shows it as “secure”, which means SSL works perfectly.
We have no clue what is going on and how we can solve this puzzle.
If you have any suggestions or assistance to guide is in setting up our SSL, it would be much appreciated. Also, please check out our dummy page located in https://old.grantlar.uz It should open without any problems AND it should have a browser-trusted SSL certificate.

We thought our domain might be blocked in Uzbekistan and upon request from the VDS host we ran MTR and NMAP tests from both ends: US based VDS to Uzbekistan based computer and vice versa. Results came out fine.

We’re utterly confused and need your help, please.


Old.grantlar.uz Error getting validation data
#2

Hi @axodjakov,

The site is working fine from my side so the problem could be a transparent proxy, a firewall issue blocking ips geo located on UZ, a dns problem pointing your domain to another ip… etc.

Could you please show the output of these commands from a computer located in UZ?.

echo | openssl s_client -connect old.grantlar.uz:443 -servername old.grantlar.uz 2>/dev/null | awk '/Certificate chain/,/---/'
echo | openssl s_client -connect old.grantlar.uz:443 -servername old.grantlar.uz 2>/dev/null | openssl x509 -noout -text

Edit: Also post the output of these commands:

host old.grantlar.uz
host old.grantlar.uz 1.0.0.1

Cheers,
sahsanu


#3

Thanks for replying @sahsanu

First command gave me this:
unable to load certificate
140584742950144:error:0906D06C:PEM routines:PEM_read_bio:no start
line:…/crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE

Others:
old.grantlar.uz has address 198.58.114.191
Using domain server:
Name: 1.0.0.1
Address: 1.0.0.1#53
Aliases:
old.grantlar.uz has address 198.58.114.191


#4

And these ones:

curl -vIL https://old.grantlar.uz

curl -vIkL https://old.grantlar.uz


#5

I wonder if Uzbekistan ISPs have started doing something like this interception system that reportedly exists in Kazakhstan?

https://news.ycombinator.com/item?id=10663843

Or it could be a site-specific block by the ISP if it doesn’t happen for other sites.

@axodjakov, could you save the certificate from a browser so we can look at it? Browsers should provide an option to save the site’s certificate as a PEM text file, although you might have to click through a few windows to get to that option.


#6

SSLLabs shows:
# TLS 1.2 (server has no preference)
Which may explain why you are having issues.
Look into setting the SSLCipherSuite order.

TLS_RSA_WITH_AES_128_CBC_SHA (0x2f) WEAK 128
TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33) DH 4096 bits FS 128
TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (0x41) WEAK 128
TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA (0x45) DH 4096 bits FS 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) ECDH sect571r1 (eq. 15360 bits RSA) FS 128
TLS_RSA_WITH_AES_128_CBC_SHA256 (0x3c) WEAK 128
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x67) DH 4096 bits FS 128
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c) WEAK 128
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e) DH 4096 bits FS 128

Or your client doesn’t support DH primes of 4096 bits.


#7

@rg305, it’s a good point that the error was ERR_SSL_PROTOCOL_ERROR rather than a certificate error, but I’m still curious about the country-specific behavior here.

@axodjakov, another thing to look at would be to run openssl s_client -connect old.grantlar.uz:443 -servername old.grantlar.uz both with and without the VPN, which will give a little bit more technical information about what the TLS handshake looks like in both circumstances. It would be very interesting to see the output.


#8

Here’s the output of running without VPN:

azamat@debian:~$ openssl s_client -connect old.grantlar.uz:443 -servername old.grantlar.uz
CONNECTED(00000003)
140100580275456:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:…/ssl/record/ssl3_record.c:252:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 5 bytes and written 200 bytes
Verification: OK

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1525019887
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no

I don’t know how to run this command with through VPN.


#9

OK, that is a serious low-level failure (suggesting that it’s not even using the TLS protocol at all). This strongly suggests ISP interference of some kind to me, given that the site works fine from elsewhere.

I tried this command from outside of Uzbekistan, and it showed a successful TLS session negotiation and key exchange with a valid Let’s Encrypt certificate presented in the handshake.

You could also try curl -v http://old.grantlar.uz:443/ just out of curiosity to see if it thinks it’s speaking HTTP on port 443.


#10

@schoen
Here’s the output of curl:

azamat@debian:~$ curl -v http://old.grantlar.uz:443/

  • Trying 198.58.114.191…
  • TCP_NODELAY set
  • Connected to old.grantlar.uz (198.58.114.191) port 443 (#0)

GET / HTTP/1.1
Host: old.grantlar.uz:443
User-Agent: curl/7.52.1
Accept: /

Object not found

  • Curl_http_done: called premature == 0
  • Connection #0 to host old.grantlar.uz left intact

#11

I think this is most likely ISP interference. :frowning: Can you see in your web server logs whether the requests originating in Uzbekistan actually reach your server?


#12

I’m amazed, my requests from Uzbekistan don’t show up in the access log. All other requests made by you and others are present. There’s the complete access log for yesterday:

199.87.154.255 - - [29/Apr/2018:15:00:52 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
51.15.86.162 - - [29/Apr/2018:15:00:54 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
185.104.120.4 - - [29/Apr/2018:15:00:56 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
198.50.200.129 - - [29/Apr/2018:15:00:58 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
5.149.248.230 - - [29/Apr/2018:15:01:00 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
46.101.161.246 - - [29/Apr/2018:15:01:02 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
66.175.208.248 - - [29/Apr/2018:15:01:09 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
192.36.27.4 - - [29/Apr/2018:15:01:13 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
94.230.208.148 - - [29/Apr/2018:15:01:24 +0000] “GET / HTTP/1.1” 200 4674 “-” “Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0”
2.152.62.63 - - [29/Apr/2018:15:17:31 +0000] “GET / HTTP/1.1” 200 4869 “SSL doesn’t resolve only in Uzbekistan” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0”
2.152.62.63 - - [29/Apr/2018:15:17:32 +0000] “GET /favicon.ico HTTP/1.1” 404 1888 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0”
2.152.62.63 - - [29/Apr/2018:15:17:32 +0000] “GET /favicon.ico HTTP/1.1” 404 1888 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0”
2.152.62.63 - - [29/Apr/2018:15:17:32 +0000] “GET /favicon.ico HTTP/1.1” 404 1888 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0”
64.41.200.105 - - [29/Apr/2018:15:17:58 +0000] “GET / HTTP/1.1” 200 4780 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0”
144.76.71.248 - - [29/Apr/2018:16:03:22 +0000] “HEAD / HTTP/1.1” 200 4102 “-” “curl/7.52.1”
144.76.71.248 - - [29/Apr/2018:16:03:28 +0000] “HEAD / HTTP/1.1” 200 4102 “-” “curl/7.52.1”
144.76.71.248 - - [29/Apr/2018:16:03:40 +0000] “HEAD / HTTP/1.1” 200 4102 “-” “curl/7.52.1”
207.55.114.218 - - [29/Apr/2018:16:07:07 +0000] “GET / HTTP/1.1” 200 4869 “SSL doesn’t resolve only in Uzbekistan” “Mozilla/5.0 (X11; Linux i686; rv:57.0) Gecko/20100101 Firefox/57.0”
207.55.114.218 - - [29/Apr/2018:16:07:07 +0000] “GET /favicon.ico HTTP/1.1” 404 1888 “-” “Mozilla/5.0 (X11; Linux i686; rv:57.0) Gecko/20100101 Firefox/57.0”
207.55.114.218 - - [29/Apr/2018:16:07:07 +0000] “GET /favicon.ico HTTP/1.1” 404 1888 “-” “Mozilla/5.0 (X11; Linux i686; rv:57.0) Gecko/20100101 Firefox/57.0”
176.61.21.220 - - [29/Apr/2018:16:09:10 +0000] “HEAD / HTTP/1.1” 200 4097 “-” “curl/7.58.0”
144.76.71.248 - - [29/Apr/2018:16:13:00 +0000] “HEAD / HTTP/1.1” 200 303 “-” “curl/7.52.1”
144.76.71.248 - - [29/Apr/2018:16:13:05 +0000] “GET / HTTP/1.1” 200 1356 “-” “curl/7.52.1”
144.76.71.248 - - [29/Apr/2018:16:13:14 +0000] “HEAD / HTTP/1.1” 200 303 “-” “curl/7.52.1”
2.152.62.63 - - [29/Apr/2018:16:15:31 +0000] “GET / HTTP/1.1” 200 4869 “-” “Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0”
75.51.0.159 - - [29/Apr/2018:16:22:42 +0000] “GET / HTTP/1.1” 200 4869 “SSL doesn’t resolve only in Uzbekistan” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299”
64.41.200.108 - - [29/Apr/2018:16:23:11 +0000] “GET / HTTP/1.1” 200 4780 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0”
4.53.111.66 - - [29/Apr/2018:17:04:54 +0000] “GET / HTTP/1.1” 200 5403 “-” “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)”
34.241.173.216 - - [29/Apr/2018:17:54:59 +0000] “GET /robots.txt HTTP/1.1” 200 343 “-” “Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)”
34.241.173.216 - - [29/Apr/2018:17:55:00 +0000] “GET /robots.txt HTTP/1.1” 200 4122 “-” “Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)”
104.34.82.118 - - [29/Apr/2018:18:08:03 +0000] “GET / HTTP/1.1” 200 4869 “SSL doesn’t resolve only in Uzbekistan” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36”
104.34.82.118 - - [29/Apr/2018:18:08:03 +0000] “GET /favicon.ico HTTP/1.1” 404 5696 “https://old.grantlar.uz/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36”
34.241.173.216 - - [29/Apr/2018:18:10:18 +0000] “GET / HTTP/1.1” 200 4659 “-” “Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)”


#13

Web server logs would only show requests after a connection was made.
Like: GET, HEAD, POST
But your having issues establishing a connection.
For those “attempts”, you would have to look at your firewall logs.

Additionally, you might want to increase the detail in the web server logging…
Try including:
%{SSL_PROTOCOL}x %{SSL_CIPHER}x


#14

This is a further indication that this could be a result of ISP interference, unfortunately.


#15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.