My web server is (include version): Apache (recent version)
The operating system my web server runs on is (include version): Windows server
I can login to a root shell on my machine (yes or no, or I don’t know): yes
I have a problem of slowness on a website that makes several hundred thousand visitors a month, and since the switch to https.
The website is hosted on a dedicated server and uses for the SSL certificate Let’s Encrypt.
The problem is the following: since the switch to https, there are very significant slowdowns at times (sometimes 20 to 30 seconds to load a page).
In the browser, the load “freezes” for several seconds and is indicated at the bottom "Establishing a TLS connection with …"
The slowdowns have no logic: it can be 1 or 2 days during the week (but not necessarily the same days), at certain times only (but not necessarily the same hours), the slowdown time also differ depending on the case, and it’s not at all related to the traffic on the site at a time T (there can be little traffic and big delays).
In doing a waterfall, we always see the same thing: it is the SSl which is the cause of these slowdowns (precisely “HTTPS handshake time” who is the only cause).
How is it possible ? Is it linked to Let’s Encrypt or a bad configuration of the certificate on the server?
How to solve this problem?
Thank you because it’s really disturbing …
The certificate itself has absolutely no effect on this - the issue would lie in the server configuration itself. Would you post your config files? That would be the easiest way to see if something is set up abnormally.
Perhaps some sort of OCSP stapling issue?
OCSP stapling was the first thing I thought of too.
Might be worth testing if you can connect directly from your server to the CDN that serves OCSP responses - currently http://ocsp.int-x3.letsencrypt.org I believe - without any unexpected delays?
Thank you for your feedback, I will look at this to provide you with the requested items.
For information, on this site (https://www.ssllabs.com/ssltest/index.html) to test the certificate, the OCSP Stapling does not seem to be activated (see screenshot) … So it would be the origin of the problem ?
That’s odd, when I checked yesterday it did seem to be activated. That might indicate an intermittent problem with it indeed.
…oops. on closer inspection, you’re looking at the OCSP must staple field, which is a different thing.
OCSP is used to check for revoked certificates. “Normally” the check is done by the browser. If you enable OCSP Stapling (which it seems you have done - see the screenshot below), the check is done by the server instead, and the result is sent as part of the TLS handshake. This is better for performance and user privacy so it’s generally a good thing, but it can occasionally cause problems if the outgoing connection from your server is flaky.
OCSP Must Staple is a separate thing; it tells browsers not to accept your certificate unless you’re using OCSP stapling. However, just because it’s disabled doesn’t mean OCSP stapling itself is disabled, just that browsers won’t complain if it is.
I’d suggest some curl or openssl commands to run to check if things are working as expected, but you mentioned you’re using Windows and I don’t know if the commands are the same there.
Ok, I understand better the distinction between the two …
I am taking your commands, we will try if possible to “adapt” if necessary for Windows
PS: what tool do you use on your screenshot?
It’s the same tool you used, just further down the page
Here’s what I’d try on a Linux server:
openssl ocsp -url http://ocsp.int-x3.letsencrypt.org -header Host=ocsp.int-x3.letsencrypt.org -no_nonce -text -issuer chain.pem -cert cert.pem
chain.pem with the actual locations of the files on your server.)
Ok for the tool, I was toot fast
We will try the commands and keep you informed…
For information, below the waterfall (i’ve forgotten to upload on my first post).
Also look at MTU issues when troubleshooting connection issues.
Here are some returns:
Regarding the command lines:
- for openSSL, it says: Error 400 (Reason: bad request)
- not easy to run the curl because Windows does not integrate basic. The curl gives nothing, no error, no confirmation (but I have a ping, so it’s supposed to be good).
I took the opportunity to regenerate the certificate, just in case the problem comes from there. I go through a windows plugin (https://github.com/Lone-Coder/letsencrypt-win-simple/releases) that I have updated to version 1.9.4 to 188.8.131.52
@Osiris: I’m not sure it could be an MTU problem because it was when we passed this site in HTTPS that the problem occurred (no problem before).
MTU problems can remain hidden until running into cryptographically involved protocols like SSL key exchange or SSH. Lowering MTU to 1384 and seeing if the problem goes away is a very simple diagnostic method to rule it out.
for openSSL, it says: Error 400 (Reason: bad request)
I think the command given to you was not quite right. Try this:
openssl ocsp -verify_other issuer.pem -issuer chain.pem -cert cert.pem -text -url http://ocsp.int-x3.letsencrypt.org -header "Host" "ocsp.int-x3.letsencrypt.org"
Ok, for the openssl command line I will test.
For MTU, I followed the preco of this article quite clear (https://serenity-networks.com/how-to-change-the-tcpip-mtu-on-windows-server-2016/)
Here are the results below (see screenshots)on the console of Windows Server. 1500 seems incorrect ? 1472 seems ok and the maximum accepted (1473 is too high = 1500).
Would that confirm that the MTU needs to be changed? And if so, can this have unwanted “side effects”?
Seems like it. You might want to check the OCSP URL (see above) too, just like you did with google.com.
1472 is not what it appears, it is excluding the 28 bytes of IP and ICMP overhead which brings it to a total of 1500.
I don't find that using ping to determine MTU is that reliable.
My suggestion would be to reduce it immediately to 1384, test if it makes any difference. If not, you can go back to 1500 and rule out MTU. If it does help, you can increase it slowly until the issue comes back.
What was the outcome of the openssl command?
@Osiris Tested with the url of the OCSP, the result is indeed the same as for google.com
@_az For the openssl command, I do not have a file corresponding to "issuer.pem", I have one for chain.pem and cert.pem, but not for issuer.pem. So, I am "stucked".
For the MTU, ok i willl try that. I'm just waiting to see further slowdowns in the coming days to do this (down MTU to 1384). For 2 days, the ssl plugin used for Windows (see link above) has been updated and the certificate renewed, so I wait to see if it solved the problem or not (to eliminate the causes one by one).
Sorry - issuer.pem is the same as chain.pem in this instance.
Here the result "Responder Error : unauthorized <6>"