Thanks, looks like you did it exactly right. I’m checking it out now … it’s looking not as straightforward as I hoped. The TLS session seems to be established and then dies shortly after.
And since Safari does not support saving the pre-master secrets, there’s no easy way to see exactly what’s happening in those encrypted data frames.
Could you also include a screenshot how Safari looks when it fails?
This is how it’s been sitting since I did the test a few minutes ago:
It basically stays on this screen for 15-20 mins or longer, then a message appears saying something along the lines of “a connection cannot be established to the server…”. I’ll send you a screen shot of that error msg too once it appears.
And everything works dandy once I turn on the VPN, be it on the Mac or iPhone.
And as I said in the start, it fails on all the iPhone browsers when the VPN is turned off. Here are screen shots:
Chrome (iPhone):
FireFox (iPhone):
Safari (iPhone):
Running the iPhone app that hits the same domain throws the following error:
Error Domain=NSURLErrorDomain Code=-1001 “The request timed out.” UserInfo={NSUnderlyingError=0x2816058c0 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 “(null)” UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, NSErrorFailingURLStringKey=https://popupjobs.com//*****, NSErrorFailingURLKey=https://popupjobs.com//*****, _kCFStreamErrorDomainKey=4, _kCFStreamErrorCodeKey=-2102, NSLocalizedDescription=The request timed out.}
Safari on my Mac is still on the same screen that I sent you.
To me, this looks like a case of intentional network filtering.
I am not an expert in this but I believe there’s a tell-tale sign in the packet captures that you’ve provided.
In the working.pcap, the IP TTL (time-to-live) in all of the packets (from the server), is 46. I think this represents the 16 hops between the client and server (64-46). The TTL remains the same in all of the IPv4 traffic - from TCP establishment, to TLS handshake, to TLS application data.
Compare that to broken.pcap. Throughout the TCP open and TLS handshake, the TTL is also 46 (as we would expect, the number of hops is the same).
However, once your client starts sending TLS application data (e.g. the HTTP request), the server begins responding with empty TCP ACK packets. What’s significant about these is that:
I do not know why this would only affect Safari. It’s possible that because Chrome was using TLS 1.3, their networking filtering doesn’t trigger for Chrome.
Or maybe there’s just some really specific flag that sets off the network filter that we can’t easily identify. As I explained earlier, all browsers/devices’ TLS traffic looks slightly different to each other.
I am also not 100% sure if my interpretations of the facts is correct. I’m not sure who else on this forum knows about this kind of stuff, maybe @mnordhoff ?
It’s possible to get foolproof evidence, and that’s if you simultaneously run a packet capture on both the server and client, and check if the server ever receives the client’s TLS application data, post-handshake.
tnx for your detailed explanations. I think I understand the core of what you're trying to explain, but most of the technical stuff kinda flew right over my head :P. As I said, networking isn't my strong suit.
Although, again, what keeps bugging me is that fact that, as far as I know, the LE certificate that gets applied by Laravel Forge seems to run into this issue. So a test domain that one of your colleagues sent me yesterday was working just fine. That's why I think it's not an issue with the certificate itself but, perhaps, something about the configuration that's done by Laravel Forge...?
I'm starting the same discussion on the Laracast forum to see if they have anything to add. Would you have any suggestions on where I should go from here?
How would I got about doing that?
Yes, it's possible. Laravel Forge configures nginx with certain security parameters (such as the ciphersuites that you were trying to configure with @rg305), and then the mutual session configuration that Safari and nginx negotiate go onto trigger the filtering response by the guilty ISP. (Assuming my theory is right).
If the nginx parameters are different in the right way, maybe the filtering response never triggers at all.
Was your colleague's site in the same datacentre as your site?
Well, you would run the Safari Wireshark test again ... but you would also run on the server at the same time (assuming your server's main network interface is eth0):
tshark -i eth0 -w server.pcap -r "host <your browser's external ip address>"
and then upload broken.pcap and server.pcap.
This is gonna sound real dumb, but could you also tell me what your server’s MTU is?
ifconfig | grep -i mtu
I have one more configuration-specific question. Can you please run:
nginx -V 2>&1 | grep -i openssl
openssl version
Since your server supports TLS 1.3, it could be possible that it’s running some dank version of openssl that has a bug in it that screws up Apple clients. But if it’s 1.1.1 or 1.1.1a (not dev!), should be fine.
If you're talking about the test domain that I mentioned that worked, I have no clue...It wasn't my colleague's. It was a URL that @rg305 PMd me for testing purposes.
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
Just ran this command in the terminal after SSHing into my server. I replaced the placeholder with the same host IP that I used on wireshark. This is what I got:
Command 'tshark' not found, but can be installed with:
sudo apt install tshark
Should I run the above sudo command?
Yeah, you can install tshark from apt. I’m going to bed soon so even if you do that test now I wouldn’t check it until much later.
Regarding your MTU, a total hail-mary to fix this issue could be:
ifconfig eth0 mtu 1300
ifconfig eth1 mtu 1300
Close and re-open Safari, and try connect. If it doesn’t help, set it back to 1500.
Don’t forget to check my question above about openssl version!
built with OpenSSL 1.1.0g 2 Nov 2017 (running with OpenSSL 1.1.1 11 Sep 2018)
OpenSSL 1.1.0g 2 Nov 2017 (Library: OpenSSL 1.1.1 11 Sep 2018)
I'll do the as you directed to see what happens. Tnx for all your assistance. Sweet dreams. 
I ran "sudo apt install tshark" followed by the following command after installation:
i got:
tshark: The file "host 162.243.165.63" doesn't exist.
What am I doing wrong?
Sorry, it’s -f instead of -r. Brain stopped working a few posts ago 
.
You need to put the IP of your browser (what it says when you visit ifconfig.me, not of your server).
also, you need to make it:
-f "host <ip> and port 443"
or else it will spam you with your own SSH packets.
lol....i think that's what was happening, cuz i put the host 162.... ip. It was capturing something, just not sure what was going on.
Now i'm running it with the IP that i got with inconfig.me
wireshark isn't capturing anything any more, but there's a capture counter that keeps incrementing on my terminal:
Capturing on 'eth0'
1328 <-- this number is incrementing
i get:
SIOCSIFMTU: Operation not permitted
That ifconfig command needs to be run with sudo because it can only be done by root.
Although the NGINX is up-to-date:
nginx version: nginx/1.15.5
built with OpenSSL 1.1.0g 2 Nov 2017 (running with OpenSSL 1.1.1 11 Sep 2018)
I wonder on seeing the two versions of OpenSSL…
Does that matter?
Should he recompile it with only 1.1.1?
I thought about this too and I'm not sure. It could cause some subtle bug with Apple devices, but this could be ruled out with the server-side packet capture.
The TTL thing really makes me think this is censorship rather than a TLS issue.
This means it's working, but the output is being written to the server.pcap file.
To verify, you can replay the contents of the file:
tshark -r server.pcap
wondering if the following error msg thrown by the iPhone app means anything to you guys?
It’s the expected behavior considering what’s happening on the network level. No inbound TLS application traffic = request times out.
The rest of the error is just Objective C-isms.
In those 88 lines I can't find how it provides x448 support as shown in SSLLabs output:
Supported Named Groups
x25519, secp256r1, x448, secp521r1, secp384r1 (server preferred order)
So, I'm even more confused now.