Accessing letsencrypt ssl sites from Iran through Apple devices

So here’s a challenging problem:

I have a domain (popupjobs.com) through GoDaddy and a server on DigitalOcean (nginx 1.15.5) that I’ve deployed through Laravel Forge (LF). I’ve also used LF to add a LetsEncrypt certificate to the domain.

Here’s what happens when trying to access the site from Iran AFTER adding the letsencrypt certificate:

Everything is fine through:

  • Chrome, Firefox, IE browsers on MacOS/Windows
  • Chrome, Firefox on Android devices
  • Android app that hits the same server/domain

The site can ONLY BE ACCESSED WITH A VPN through:

  • Safari browser on MacOS/Windows
  • all browsers on the iPhone (iOS 11/12)
  • iOS app that hits the same server/domain

In other words, Apple related devices/browsers/apps can only connect to the server through a VPN. The connection times out without a VPN.

Now if this were a sanctions related issue, then I’d expect to see the same problem through all devices/browsers. But the fact that this issue only arises with some Apple related devices/browsers leads me to believe that the issue is config related. In other words, there is something about the letsencrypt config that Apple doesn’t like when trying to access the server from Iranian ISPs.

I found what turned out to be a temporary fix for the iPhone app (creating default instead of shared nsurlsessions), but even that stopped working a couple of days ago.

Any assistance in helping resolve this issue is greatly appreciated. My iOS users are dropping like flies since they can’t run the app without a VPN.

Here’s a copy of my Nginx config file:

include forge-conf/popupjobs.com/before/*;

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name popupjobs.com;
root /home/forge/popupjobs.com/public;

# if ($scheme = https) {
#     return 301 http://$server_name$request_uri;
# }
# FORGE SSL (DO NOT REMOVE!)
ssl_certificate /etc/nginx/ssl/popupjobs.com/******/server.crt;
ssl_certificate_key /etc/nginx/ssl/popupjobs.com/******/server.key;

#ADDED Improve HTTPS performance with session resumption
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
	
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/dhparams.pem;

#ADDED Enable HSTS (https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security)
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";  

#ADDED Enable OCSP stapling (http://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox)
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/popupjobs.com/458996/server.crt;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";

index index.html index.htm index.php;

charset utf-8;

# FORGE CONFIG (DO NOT REMOVE!)
include forge-conf/popupjobs.com/server/*;

location / {
    try_files $uri $uri/ /index.php?$query_string;
}

location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt  { access_log off; log_not_found off; }

access_log off;
error_log  /var/log/nginx/popupjobs.com-error.log error;

error_page 404 /index.php;

location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/var/run/php/php7.1-fpm.sock;
    fastcgi_index index.php;
    include fastcgi_params;
}

location ~ /\.(?!well-known).* {
    deny all;
}

}

FORGE CONFIG (DO NOT REMOVE!)

include forge-conf/popupjobs.com/after/*;

Have you tried doing a packet capture on either side of a failing connection? Seeing if the ClientHello and ServerHello arrive unmolested? Compare the messages with those of successful connections?

Should be simple to perform this analysis from macOS and Windows.

Have you ruled out IPv4/IPv6 as a factor?

I wouldn't assume that. All the browsers run different TLS stacks, so their TLS handshakes all look slightly different. They even handle IPv4/IPv6 in subtlety different ways. Something about Apple's might trigger a filtering response.

1 Like

Thanks for your response az. Haven’t tried the capture. I’m a web/app developer and a networking dummy :). Will google to fine out how to perform the capture, but if there’s an easy way of performing that I’m all ears.

And how would I go about ruling out IPv4/6 factors?

You could try temporarily withdrawing the IPv6 record for your domain and seeing if it helps.

Downloading Wireshark for macOS should get you started on your packet capture journey. (Assuming you are inside Iran and can reproduce the issue yourself).

Great, thanks! Will do as you suggested to see what happens.

You might want to look into reordering these:
Supported Named Groups
x25519, secp256r1, x448, secp521r1, secp384r1

Hello and thanks for your response, rg. I’ve seen those named groups when I tested the domain on ssllabs, but I have no clue what they are or how to go about manipulating them.

The current order is the “server preferred order” as noted by ssllabs. Are you saying that I should just reorder them in different combinations to see which works, or is there a certain order you have in mind?

Lookup: ssl_ecdh_curve
I prefer to use:
ssl_ecdh_curve sect571r1:secp521r1:sect409r1:secp384r1:sect283r1:prime256v1;

Yes, strongest to weakest.
And maybe drop those that are not recommended.

@rg305 will look into it. Tnx! :slight_smile:

1 Like

@_az I disabled IPv6 on my server, but to no avail. Been trying to figure out my way around Wireshark now. For many URLs, it seems like once they get loaded in the browser they get cached in Wireshark and the http requests no longer get displayed. I keep googling for a solution but still haven’t figured out the problem. Am I missing something.

@rg305 I added the ssl curve you posted to my nginx config file (that’s what I was supposed to do, right?), but no success.

Yes, but it seems you may not have added it in the right place.
I would put it with these lines:

@rg305 I’d previously placed it under “resolver_timeout”. Just placed it under “ssl_dhparam”, but still no luck.

Please show the exact line used.
And did you restart or reload the web service?
If you have more then one vhost vconfig, be sure it is in the one with:
server_name popupjobs.com;

Also show:
openssl version
openssl ecparam -list_curves
nginx -V

@rg305 I restarted everything (server, nginx, php) after saving the config. I’m using the Laravel Forge panel to edit the config file and restart everything.

opensslversion: OpenSSL 1.1.0g 2 Nov 2017 (Library: OpenSSL 1.1.1 11 Sep 2018)
nginx v: nginx/1.15.5
openssl ecparam -list_curve: (that’s a lot of text. Are you looking for a specific section, or should I post the entire thing?

Here’s the config:

include forge-conf/popupjobs.com/before/*;

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name popupjobs.com;
root /home/forge/popupjobs.com/public;

# if ($scheme = https) {
#     return 301 http://$server_name$request_uri;
# }
# FORGE SSL (DO NOT REMOVE!)
ssl_certificate /etc/nginx/ssl/popupjobs.com/458996/server.crt;
ssl_certificate_key /etc/nginx/ssl/popupjobs.com/458996/server.key;

#ADDED Improve HTTPS performance with session resumption
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
	
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/dhparams.pem;
ssl_ecdh_curve sect571r1:secp521r1:sect409r1:secp384r1:sect283r1:prime256v1;



#ADDED Enable HSTS (https://developer.mozilla.org/en-US/docs/Security/HTTP_Strict_Transport_Security)
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";  


#ADDED Enable OCSP stapling (http://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox)
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/popupjobs.com/458996/server.crt;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;



add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";

index index.html index.htm index.php;

charset utf-8;

# FORGE CONFIG (DO NOT REMOVE!)
include forge-conf/popupjobs.com/server/*;

location / {
    try_files $uri $uri/ /index.php?$query_string;
}

location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt  { access_log off; log_not_found off; }

access_log off;
error_log  /var/log/nginx/popupjobs.com-error.log error;

error_page 404 /index.php;

location ~ \.php$ {
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/var/run/php/php7.1-fpm.sock;
    fastcgi_index index.php;
    include fastcgi_params;
}

location ~ /\.(?!well-known).* {
    deny all;
}

}

FORGE CONFIG (DO NOT REMOVE!)

include forge-conf/popupjobs.com/after/*;

You can upload it as a txt file.

So far, everything looks as I would expect it...
Yet SSLLabs still shows:
Supported Named Groups
x25519, secp256r1, x448, secp521r1, secp384r1 (server preferred order)

Please also show:
nginx -V
grep -Ri 'ssl_ecdh_curve' /etc/nginx

What is that for? What does that do?

@rg305 It's a part of the default nginx config file generated by LF. The ssl directives get added to this file when you request the LE certificate for the domain through the LF panel.

Please also show:
nginx -V
grep -Ri 'ssl_ecdh_curve' /etc/nginx

openssl ecparam -list_curve
[You can upload it as a txt file.]

@rg305

nginx version: nginx/1.15.5

built with OpenSSL 1.1.0g 2 Nov 2017 (running with OpenSSL 1.1.1 11 Sep 2018)

TLS SNI support enabled

configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-FDSBVO/nginx-1.15.5=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-FDSBVO/nginx-1.15.5/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-FDSBVO/nginx-1.15.5/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-FDSBVO/nginx-1.15.5/debian/modules/http-echo --add-dynamic-module=/build/nginx-FDSBVO/nginx-1.15.5/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-FDSBVO/nginx-1.15.5/debian/modules/http-subs-filter

@rg305

/etc/nginx/sites-enabled/popupjobs.com: ssl_ecdh_curve sect571r1:secp521r1:sect409r1:secp384r1:sect283r1:prime256v1;
/etc/nginx/sites-available/popupjobs.com: ssl_ecdh_curve sect571r1:secp521r1:sect409r1:secp384r1:sect283r1:prime256v1;