Sudden increase in TLS handshake time

Since a few days getting tons of TLS handshake timeouts.
For many users it’s taking over 60 seconds (!) to perform TLS handshake.
I suspect the Let’s Encrypt CRL servers are not responsive.
Is there any way to debug/fix this? I’m really at a loss here not able to work around this. Domain is newtis.info

1 Like

You really should look into your server performance first. Giving us the required info would help a lot:

Server Operating System:
Can you directly log into a root shell/Windows rdp?
Are you using a control panel (cPanel/Plesk/etc)?

I will say that it all came up quickly for me:
:confused: $ curl -ILv newtis.info
* Rebuilt URL to: newtis.info/
* Trying 2a01:4f9:2b:16c4::228…
* TCP_NODELAY set
* Connected to newtis.info (2a01:4f9:2b:16c4::228) port 80 (#0)
> HEAD / HTTP/1.1
> Host: newtis.info
> User-Agent: curl/7.58.0
> Accept: /
>
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Date: Tue, 25 Feb 2020 13:11:26 GMT
Date: Tue, 25 Feb 2020 13:11:26 GMT
< Server: Apache
Server: Apache
< Location: https://www.newtis.info/
Location: https://www.newtis.info/
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1

<
* Connection #0 to host newtis.info left intact
* Issue another request to this URL: 'https://www.newtis.info/'
*   Trying 2a01:4f9:2b:16c4::228...
* TCP_NODELAY set
* Connected to www.newtis.info (2a01:4f9:2b:16c4::228) port 443 (#1)* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /system/etc/security/cacerts
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=newtis.info
*  start date: Jan 19 11:32:05 2020 GMT
*  expire date: Apr 18 11:32:05 2020 GMT
*  subjectAltName: host "www.newtis.info" matched cert's "www.newtis.info"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: www.newtis.info
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 25 Feb 2020 13:11:28 GMT
Date: Tue, 25 Feb 2020 13:11:28 GMT
< Server: Apache
Server: Apache
< Last-Modified: Fri, 08 Mar 2019 20:47:09 GMT
Last-Modified: Fri, 08 Mar 2019 20:47:09 GMT
< ETag: "bbc-5839b53950d40"
ETag: "bbc-5839b53950d40"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 3004
Content-Length: 3004
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Strict-Transport-Security: max-age=31536000
Strict-Transport-Security: max-age=31536000
< Content-Type: text/html
Content-Type: text/html

<
* Connection #1 to host www.newtis.info left intact
:/ $
1 Like

Server performance is fine, as well as the network (1Gbit). No bottlenecks. No cpanel.

Hi,

Let's Encrypt does not use CRL.... Let's Encrypt utilized OCSP and your web server should configures it to automatically staple the OCSP response from Let's Encrypt (so every client doesn't need to make a request to Lets Encrypt's server)

I'm not sure why your client would have a slow connection time, but generally it's not related to OCSP or CRL (because Chrome doesn't even check for it), sometimes it's just ... slow in general.
Also slow connection could happen in few different stages, including SSL handshake (but generally in TTFB).

As @zetarevan said, server performance might be the issue, not Let's Encrypt's server.

Thank you

1 Like

Server performance is fine, as well as the network (1Gbit). No bottlenecks.

What do you use to monitor this performance? I highly suggest prometheus and the node_exporter, but it's a bit more advanced to get set up. For the typical systems administrator, starting with the sysstat tool sar is pretty good too.

During times of slow handshakes can you run the following

mtr -c 20 acme-v02.api.letsencrypt.org --report
netstat -tupan
sar -u
sar -R
1 Like

Checking the web server it does not seem to provide a stapled OCSP response, This could be the source of additional latency, forcing the browser to go request and wait for one.

@RustyX You should look at the documentation for your webserver software on how to enable this

EDIT: I see you are using a DigiCert certificate right now, unfortunately the page still took about a second to load for me initially, on subsequent reloads however it was nearly instant. Which I suspect is related to stapling.

2 Likes

I think the performance issue is really depends on the browser user uses. If they use chrome, there shouldn’t be an need for the OCSP fetch (since they didn’t check for that). User might still experience performance issues when they use chrome regardless. (But in any sense, my Firefox didn’t have any delay opening his website… )

1 Like

Yes I switched to DigiCert and that cut the handshake timeouts by a factor of 3, but didn’t eliminate the delay entirely. I analyzed the user-agents of the IPs that were experiencing timeouts and was unable to pinpoint a specific user-agent. The vast majority were Chrome on Windows (45%) and Android (40%) though there was also some percentage of Trident, Firefox, iPhone and others.

What definitely helped was OCSP stapling and HTTP/2. I guess it should work the same with Let’s Encrypt. Will test when I get a chance. :+1:

1 Like

I ran that curl test on an android terminal, btw. But I think it’s more your client base. Might also want to break down your results by the region of the source IPs.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.