Problems of significant slowness on a website since passing in https with let's Encrypt

_az · December 1, 2017, 9:45pm

Seems like your server is running well now, fast handshake even with OCSP enabled. Did you change anything/can you confirm it is fixed?

Matt69 · December 4, 2017, 9:56am

Indeed it works well for the moment, but i have to wait several days to confirm it because it can have days without any problems… and suddenly they reappears… !

The only things we’ve done for the moment is to renew the certificate and update the windows module for Let’s Encrypt (https://github.com/Lone-Coder/letsencrypt-win-simple/releases).

What about the result of your command ? Is it normal (“Responder error…”) ?

davidfavor · December 4, 2017, 2:01pm

Wow… Using Windows will make fixing speed issues complex.

Your cert has nothing to do with site speed.

Likely first step, switch to an easy to debug runtime Distro like Ubuntu + then fix these problems.
You’re running HTTP1.1

Fix: Use HTTP2

SSLLabs tester reports incorrect chain order, so it’s likely your certs are configured incorrectly.

Fix your SSL config so SSLLabs message “Incorrect order, Extra certs” no longer emits.

You’re not running HSTS, so all your first site connections (each page view) will be slow to start.

Fix: Use HSTS.

You’re running a recent version of Apache + PHP.

Ensure you’re running MPM mod_event + PHP FPM for best speed.

Change or tune your hosting.

WebPageTest of your site reports very slow speed. Look at asset #2, a small CSS file takes a long time to download.

Change or tune your hosting.

Enable Apache level compression.

Tune your CMS (Contao) to correctly use caching.

Looks like either Contao caching is weak/broken.

Compare your site’s first few assets (which take multiple seconds to server) with a Well tuned WordPress site which serves the first asset (HTML part of site) in 290ms (half of this is DNS).

Then all other files serve simultaneously (HTTP2) + quickly (caching working correctly).

A big problem with non-WordPress CMS systems is tuning WordPress is a know set of steps. Tuning non-WordPress CMS systems requires you to personally come up with 12+ years (since first WordPress release) of tuning intelligence on your own.

bytecamp · December 4, 2017, 2:18pm

Could you elaborate on this? How is strict transport security related to loading speed?

_az · December 4, 2017, 10:55pm

Something is not right here, but I'm not sure whether or not it's perhaps due to you having the wrong input files.

I will paste both issuer.pem and cert.pem here as well as the OpenSSL command. If you still get errors for that then I would suggest you need to tag one of the Boulder engineers on the forum.

openssl ocsp -verify_other issuer.pem -issuer issuer.pem -cert cert.pem -text -url http://ocsp.int-x3.letsencrypt.org -header "Host" "ocsp.int-x3.letsencrypt.org"

WARNING: no nonce in response
Response verify OK
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
          Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
          Serial Number: 03DE11639DB463B11965819869C613D99A37
    Request Extensions:
        OCSP Nonce: 
            0410A8CED88DEC8CF9F3CACB4E223905CFA3
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
    Produced At: Dec  3 08:19:00 2017 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
      Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
      Serial Number: 03DE11639DB463B11965819869C613D99A37
    Cert Status: good
    This Update: Dec  3 08:00:00 2017 GMT
    Next Update: Dec 10 08:00:00 2017 GMT

    Signature Algorithm: sha256WithRSAEncryption
         2b:fa:e2:1d:87:04:3c:77:0a:4d:0e:0d:c1:3e:78:0d:56:ea:
         cc:1b:d8:68:35:f9:b5:a0:a6:97:27:05:a4:87:8b:a1:95:cc:
         c1:3d:2a:87:ec:99:9f:54:a1:04:d7:04:52:4d:26:84:4d:00:
         ea:54:b0:c1:7b:23:b0:c0:8f:8b:ad:a6:78:7e:0c:d5:eb:c0:
         46:8f:92:02:c2:9e:fa:98:62:b6:9e:a3:58:0c:6b:45:9a:eb:
         65:1c:93:47:55:dd:a9:3a:0d:08:d9:42:cc:d8:f8:eb:90:5e:
         f0:53:44:fb:ff:95:48:44:14:24:5c:de:e3:f8:19:46:d4:b1:
         8c:5f:dc:52:bd:44:f8:1d:8c:98:aa:80:49:ee:05:82:d2:41:
         2a:8d:f2:94:8b:9a:ea:4b:da:16:28:ed:20:71:33:34:bb:03:
         fd:25:c2:44:11:8e:cb:7b:7d:86:04:a8:bd:f4:2a:d5:81:2c:
         29:3a:74:59:4b:4d:a7:33:f4:13:64:86:17:52:75:3c:63:3e:
         00:9a:5c:36:2f:19:5b:07:23:74:ce:5a:ab:ec:06:f6:ab:1d:
         42:9b:dc:8b:68:7f:39:54:74:22:8c:a4:6c:49:e1:da:57:f6:
         1b:47:89:2d:88:f1:0a:9e:60:c1:94:ff:89:c6:d0:45:b9:09:
         a6:f2:52:6f
cert.pem: good
	This Update: Dec  3 08:00:00 2017 GMT
	Next Update: Dec 10 08:00:00 2017 GMT

cert.pem:

issuer.pem:

Matt69 · December 5, 2017, 3:01pm

@ davidfavor The same as Bytecamp for me (HSTS) … And globally, ok for your recommendations (http1.1 …) but that does not change what was in place before. So even though it may improve the overall load time, it does not have to be that which creates strong slowdowns at random times.

@_az The problem is the same. I do not know who these engineers are …

After perhaps this is only a problem related to the server (ssl would be indirectly involved as it passes through the OCSP, hence the waterfall). Site traffic (+600,000 pages viewed per month) increased significantly as slowdowns occurred. Maybe a multi-threaded php management problem (even Apache), so that the dedicated server (8threads = 2x4) can not succeed to manage as many queries at once at certain times, if not configured properly for multi threading …

jmorahan · December 5, 2017, 7:29pm

Yeah I’d imagine that if it was just general server slowness, it would show up as increased wait time rather than only handshake time, that was why I suspected it might be OCSP…

Does the openssl ocsp command display the error more or less immediately, or after a delay?

Do you have outgoing IPv6 connectivity from your server? Not sure it’s an issue if you don’t, just wondering if your webserver might be trying to contact the OCSP responder URL via IPv6 and failing?

_az · December 5, 2017, 8:55pm

Could you show full output of the openssl command? Maybe we can ask @cpu or somebody else about it.

If I were you I'd totally disable OCSP stapling if you can't get reliable successful OCSP queries to Let's Encrypt.

Matt69 · December 6, 2017, 10:39am

@jmorahan @_az

The command displays the result immediately.

For IPV6, like that I do not know but it would seem strange …?

For the OCSP, I do not think that this is the source of the problem because when the slowdown began to appear, we have 2 or 3 weeks later activated the OCSP (and it has not changed anything). The problem was prior to the establishment of the OCSP.

For the complete output of the command, it’s exactly like the one you put above in the post 25 (except the keys “Issuer Key Hash” and “OCSP Nonce” which differ …). And below “OCSP Nonce”, there is only the line “Responder Error: unauthorized <6>”.

pfg · December 6, 2017, 12:36pm

Could you try enabling mod_status and check the status page the next time you’re experiencing this issue? (Make sure you lock access down to your network or put it behind a login.)

Not sure which MPM you’re using, but I would start investigating more general apache performance bottlenecks, like a lack of available worker processes. The status page should be able to help you figure out where the requests are stuck.

If you haven’t yet, check the apache error log, events like reaching MaxRequestWorkers should cause some noise there.

bytecamp · December 6, 2017, 1:28pm

Another source of information could be Apache's error logfile (/var/log/apache2/error.log or /var/log/httpd-error.log). The webserver actually emits warnings if there are too few processes/threads to handle the incoming requests.

cpu · December 6, 2017, 2:05pm

I agree it would be useful to see the exact command you ran. Typically I associated this error with an OCSP query that didn't set the Host header correctly.

Matt69 · December 6, 2017, 4:18pm

@cpu
It’s exactly the same command as @_az has indicated :

openssl ocsp -verify_other issuer.pem -issuer issuer.pem -cert cert.pem -text -url http://ocsp.int-x3.letsencrypt.org -header "Host" "ocsp.int-x3.letsencrypt.org"

The result of this command is exactly the same than @_az (except the keys “Issuer Key Hash” and “OCSP Nonce” which differ …). And below “OCSP Nonce”, there is only the line “Responder Error: unauthorized <6>”.

Matt69 · December 6, 2017, 5:24pm

@pfg @bytecamp

Ok for modstatus, i’ll try to see it when the site will be in a slowness state… (which is not currently the case).

For the logs, here below what I have mainly seen on the following 2 files (i have deleted sometimes private informations or replace by “xxx”) :

Error logs file :

a. Errors that come up frequently, but i don’t kow if it’s specific to a slowness moment

[Mon Nov 20 16:48 2017] [mpm_winnt:notice] [pid 14692:tid 392] AH00354 : Child: Starting 64 worker threads.
[Mon Nov 20 16:48 2017] [mpm_winnt:error] [pid 14692:tid 2212] AH00326: Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
[Wed Nov 29 11:08 2017] [mpm_winnt:notice] [pid 8980:tid 472] AH00428: Parent: child process 17552 exited with status 255 – Restarting.
[Tue Nov 28 14 2017] [authz_core:error] [pid 18820:tid 948] [client] AH01630: client denied by server configuration: C:/xxx

b. Errors a day we noticed very big slowness moments for several hours

[Wed Oct 11 12:34 2017] [core:error] [pid 2368:tid 800] (20024)The given path is misformatted or contained invalid characters: [client ] AH00127: Cannot map GET /assets/html5shiv/3.7.2/html5shiv-printshiv.js"%3Cimg%20src=xyz%20OnErRor=prompt(940844)%3E HTTP/1.1 to file, referer: website:xxx/
[Mon Sep 25 10:35 2017] [access_compat:error] [pid 5944:tid 928] [client ] AH01797: client denied by server configuration: C:/xxx

SSL error logs file :

[Mon Dec 04 12 2017] [ssl:error] [pid 18584:tid 392] AH02604: Unable to configure certificate www.example.com:443:0 for stapling
[Mon Dec 04 12:40 2017] [ssl:warn] [pid 18584:tid 392] AH01906: www.example.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Dec 04 12:40 2017] [ssl:warn] [pid 18584:tid 392] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Mon Dec 04 12:40 2017] [ssl:error] [pid 18584:tid 392] AH02217: ssl_stapling_init_cert: can’t retrieve issuer certificate! [subject: emailAddress=xxx,CN=xxx,O=xxx,L=xxx,ST=France,C=FR / issuer: emailAddress=xxx,CN=x,O=xxx,L=xxx,ST=France,C=FR / serial: xxx / notbefore: Sep 5 22:22:10 2017 GMT / notafter: Aug 12 22:22:10 2117 GMT]

bytecamp · December 7, 2017, 8:55am

That's what I meant. First you should read the documentation on the MPMs of Apache and then carefully adjust the knobs.

Matt69 · December 7, 2017, 6:12pm

Ok, il will see that…

system · January 6, 2018, 6:12pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
We use HTTPS to delay access for more than 15 seconds. What's the problem? Help	7	1361	May 21, 2020
Slow (20 seconds) initial load with SSL Server	8	10686	January 24, 2017
Slow TLS Handshakes since 07 September 2022 Help	9	1916	December 24, 2022
HTTPS very slow after certificate renewal Help	11	4583	February 13, 2020
Letsencrypt OCSP response times measured? Issuance Tech	13	5684	June 15, 2016

Problems of significant slowness on a website since passing in https with let's Encrypt

Related topics