Problems of significant slowness on a website since passing in https with let's Encrypt


#21

Seems like your server is running well now, fast handshake even with OCSP enabled. Did you change anything/can you confirm it is fixed?


#22

Indeed it works well for the moment, but i have to wait several days to confirm it because it can have days without any problems… and suddenly they reappears… !

The only things we’ve done for the moment is to renew the certificate and update the windows module for Let’s Encrypt (https://github.com/Lone-Coder/letsencrypt-win-simple/releases).

What about the result of your command ? Is it normal (“Responder error…”) ?


#23

Wow… Using Windows will make fixing speed issues complex.

Your cert has nothing to do with site speed.

  1. Likely first step, switch to an easy to debug runtime Distro like Ubuntu + then fix these problems.

  2. You’re running HTTP1.1

Fix: Use HTTP2

  1. SSLLabs tester reports incorrect chain order, so it’s likely your certs are configured incorrectly.

Fix your SSL config so SSLLabs message “Incorrect order, Extra certs” no longer emits.

  1. You’re not running HSTS, so all your first site connections (each page view) will be slow to start.

Fix: Use HSTS.

  1. You’re running a recent version of Apache + PHP.

Ensure you’re running MPM mod_event + PHP FPM for best speed.

  1. Change or tune your hosting.

WebPageTest of your site reports very slow speed. Look at asset #2, a small CSS file takes a long time to download.

  1. Change or tune your hosting.

Enable Apache level compression.

  1. Tune your CMS (Contao) to correctly use caching.

Looks like either Contao caching is weak/broken.

Compare your site’s first few assets (which take multiple seconds to server) with a Well tuned WordPress site which serves the first asset (HTML part of site) in 290ms (half of this is DNS).

Then all other files serve simultaneously (HTTP2) + quickly (caching working correctly).

A big problem with non-WordPress CMS systems is tuning WordPress is a know set of steps. Tuning non-WordPress CMS systems requires you to personally come up with 12+ years (since first WordPress release) of tuning intelligence on your own.


#24

Could you elaborate on this? How is strict transport security related to loading speed?


#25

Something is not right here, but I’m not sure whether or not it’s perhaps due to you having the wrong input files.

I will paste both issuer.pem and cert.pem here as well as the OpenSSL command. If you still get errors for that then I would suggest you need to tag one of the Boulder engineers on the forum.

openssl ocsp -verify_other issuer.pem -issuer issuer.pem -cert cert.pem -text -url http://ocsp.int-x3.letsencrypt.org -header "Host" "ocsp.int-x3.letsencrypt.org"

WARNING: no nonce in response
Response verify OK
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
          Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
          Serial Number: 03DE11639DB463B11965819869C613D99A37
    Request Extensions:
        OCSP Nonce: 
            0410A8CED88DEC8CF9F3CACB4E223905CFA3
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
    Produced At: Dec  3 08:19:00 2017 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: 7EE66AE7729AB3FCF8A220646C16A12D6071085D
      Issuer Key Hash: A84A6A63047DDDBAE6D139B7A64565EFF3A8ECA1
      Serial Number: 03DE11639DB463B11965819869C613D99A37
    Cert Status: good
    This Update: Dec  3 08:00:00 2017 GMT
    Next Update: Dec 10 08:00:00 2017 GMT

    Signature Algorithm: sha256WithRSAEncryption
         2b:fa:e2:1d:87:04:3c:77:0a:4d:0e:0d:c1:3e:78:0d:56:ea:
         cc:1b:d8:68:35:f9:b5:a0:a6:97:27:05:a4:87:8b:a1:95:cc:
         c1:3d:2a:87:ec:99:9f:54:a1:04:d7:04:52:4d:26:84:4d:00:
         ea:54:b0:c1:7b:23:b0:c0:8f:8b:ad:a6:78:7e:0c:d5:eb:c0:
         46:8f:92:02:c2:9e:fa:98:62:b6:9e:a3:58:0c:6b:45:9a:eb:
         65:1c:93:47:55:dd:a9:3a:0d:08:d9:42:cc:d8:f8:eb:90:5e:
         f0:53:44:fb:ff:95:48:44:14:24:5c:de:e3:f8:19:46:d4:b1:
         8c:5f:dc:52:bd:44:f8:1d:8c:98:aa:80:49:ee:05:82:d2:41:
         2a:8d:f2:94:8b:9a:ea:4b:da:16:28:ed:20:71:33:34:bb:03:
         fd:25:c2:44:11:8e:cb:7b:7d:86:04:a8:bd:f4:2a:d5:81:2c:
         29:3a:74:59:4b:4d:a7:33:f4:13:64:86:17:52:75:3c:63:3e:
         00:9a:5c:36:2f:19:5b:07:23:74:ce:5a:ab:ec:06:f6:ab:1d:
         42:9b:dc:8b:68:7f:39:54:74:22:8c:a4:6c:49:e1:da:57:f6:
         1b:47:89:2d:88:f1:0a:9e:60:c1:94:ff:89:c6:d0:45:b9:09:
         a6:f2:52:6f
cert.pem: good
	This Update: Dec  3 08:00:00 2017 GMT
	Next Update: Dec 10 08:00:00 2017 GMT

cert.pem:

-----BEGIN CERTIFICATE-----
MIIFEDCCA/igAwIBAgISA94RY520Y7EZZYGYacYT2Zo3MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xNzExMzAwNzE4NThaFw0x
ODAyMjgwNzE4NThaMBgxFjAUBgNVBAMTDXByZXZpc3NpbWEuZnIwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCUl1J/24l6MwlBj7r/QwhdIjRweXi9ZLEb
wOoYz5WqzoMdPJRT2lYtvQ5c6WuSktin/5zkzFYdh2h93zrzCb3pHwPyf7ioH1xT
VWn6zIl9ZbqXZxSFHCNd7j65uHnxu5cGC+g6RUYoeTU5uPBGwuj3xVMcFnGYExjJ
1ovV1Fk7HS/+142qd27ePRn3nzNCSRWl3uEah+kkVFWXV7nbEbElPpRRCrHkCFRn
Q5dwxMUadi09vYhDtIBkkc3qNwXANUpJzxBPhFiHpEg7WwTHkKEclUyzusv+9Vw4
VkqpXRBvybEDJBsucAXO2N8E0tsbu8ll31o0sUvm2jDmDJaMHPKhAgMBAAGjggIg
MIICHDAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFDyjCkjrLF7076aekRmfRc6Hsgjt
MB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZFZe/zqOyhMG8GCCsGAQUFBwEBBGMw
YTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3AuaW50LXgzLmxldHNlbmNyeXB0Lm9y
ZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQuaW50LXgzLmxldHNlbmNyeXB0Lm9y
Zy8wKwYDVR0RBCQwIoINcHJldmlzc2ltYS5mcoIRd3d3LnByZXZpc3NpbWEuZnIw
gf4GA1UdIASB9jCB8zAIBgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsG
AQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIw
gZ4MgZtUaGlzIENlcnRpZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5
IFJlbHlpbmcgUGFydGllcyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhl
IENlcnRpZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0
Lm9yZy9yZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEAPJKbS5ujJXjDpemr
R30ysd+s6G9YCyELB9EtM4x9PPcM8jYG7PY6VE8boHZ/BpkidPoOv3pNfgr4HtPY
g2ItI3JGpPvQkupEHBGG12Y50S1xLNz09ikPEub1rLadhN/4wXfxW6MURgAndkGC
ioqJSxVQEw21wwJdfjyf3rsuex871Cf5Ul40mFHsmJQXHeXpshQkLWdWBYmth0k5
3I3xqMjC+xH/9fRk+jQZ/LaUltBOJ5nBHz/Zg5ifG+b5A+qeKfN/iqzO8k9MyUoQ
w7FAJxX4bzvvSn/5XotrYhlPpE2d7F2unobkp6HcC3+aMbOEScK0hNRrK/RyRxLa
4vifmw==
-----END CERTIFICATE-----

issuer.pem:

-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----

#26

@ davidfavor The same as Bytecamp for me (HSTS) … And globally, ok for your recommendations (http1.1 …) but that does not change what was in place before. So even though it may improve the overall load time, it does not have to be that which creates strong slowdowns at random times.

@_az The problem is the same. I do not know who these engineers are …

After perhaps this is only a problem related to the server (ssl would be indirectly involved as it passes through the OCSP, hence the waterfall). Site traffic (+600,000 pages viewed per month) increased significantly as slowdowns occurred. Maybe a multi-threaded php management problem (even Apache), so that the dedicated server (8threads = 2x4) can not succeed to manage as many queries at once at certain times, if not configured properly for multi threading …


#27

Yeah I’d imagine that if it was just general server slowness, it would show up as increased wait time rather than only handshake time, that was why I suspected it might be OCSP…

Does the openssl ocsp command display the error more or less immediately, or after a delay?

Do you have outgoing IPv6 connectivity from your server? Not sure it’s an issue if you don’t, just wondering if your webserver might be trying to contact the OCSP responder URL via IPv6 and failing?


#28

Could you show full output of the openssl command? Maybe we can ask @cpu or somebody else about it.

If I were you I’d totally disable OCSP stapling if you can’t get reliable successful OCSP queries to Let’s Encrypt.


#29

@jmorahan @_az

The command displays the result immediately.

For IPV6, like that I do not know but it would seem strange …?

For the OCSP, I do not think that this is the source of the problem because when the slowdown began to appear, we have 2 or 3 weeks later activated the OCSP (and it has not changed anything). The problem was prior to the establishment of the OCSP.

For the complete output of the command, it’s exactly like the one you put above in the post 25 (except the keys “Issuer Key Hash” and “OCSP Nonce” which differ …). And below “OCSP Nonce”, there is only the line “Responder Error: unauthorized <6>”.


#30

Could you try enabling mod_status and check the status page the next time you’re experiencing this issue? (Make sure you lock access down to your network or put it behind a login.)

Not sure which MPM you’re using, but I would start investigating more general apache performance bottlenecks, like a lack of available worker processes. The status page should be able to help you figure out where the requests are stuck.

If you haven’t yet, check the apache error log, events like reaching MaxRequestWorkers should cause some noise there.


#31

Another source of information could be Apache’s error logfile (/var/log/apache2/error.log or /var/log/httpd-error.log). The webserver actually emits warnings if there are too few processes/threads to handle the incoming requests.


#32

I agree it would be useful to see the exact command you ran. Typically I associated this error with an OCSP query that didn’t set the Host header correctly.


#33

@cpu
It’s exactly the same command as @_az has indicated :

openssl ocsp -verify_other issuer.pem -issuer issuer.pem -cert cert.pem -text -url http://ocsp.int-x3.letsencrypt.org -header "Host" "ocsp.int-x3.letsencrypt.org"

The result of this command is exactly the same than @_az (except the keys “Issuer Key Hash” and “OCSP Nonce” which differ …). And below “OCSP Nonce”, there is only the line “Responder Error: unauthorized <6>”.


#34

@pfg @bytecamp

Ok for modstatus, i’ll try to see it when the site will be in a slowness state… (which is not currently the case).

For the logs, here below what I have mainly seen on the following 2 files (i have deleted sometimes private informations or replace by “xxx”) :

  1. Error logs file :

a. Errors that come up frequently, but i don’t kow if it’s specific to a slowness moment

  • [Mon Nov 20 16:48 2017] [mpm_winnt:notice] [pid 14692:tid 392] AH00354 : Child: Starting 64 worker threads.

  • [Mon Nov 20 16:48 2017] [mpm_winnt:error] [pid 14692:tid 2212] AH00326: Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting

  • [Wed Nov 29 11:08 2017] [mpm_winnt:notice] [pid 8980:tid 472] AH00428: Parent: child process 17552 exited with status 255 – Restarting.

  • [Tue Nov 28 14 2017] [authz_core:error] [pid 18820:tid 948] [client] AH01630: client denied by server configuration: C:/xxx

b. Errors a day we noticed very big slowness moments for several hours

  • [Wed Oct 11 12:34 2017] [core:error] [pid 2368:tid 800] (20024)The given path is misformatted or contained invalid characters: [client ] AH00127: Cannot map GET /assets/html5shiv/3.7.2/html5shiv-printshiv.js"%3Cimg%20src=xyz%20OnErRor=prompt(940844)%3E HTTP/1.1 to file, referer: website:xxx/

  • [Mon Sep 25 10:35 2017] [access_compat:error] [pid 5944:tid 928] [client ] AH01797: client denied by server configuration: C:/xxx

  1. SSL error logs file :
  • [Mon Dec 04 12 2017] [ssl:error] [pid 18584:tid 392] AH02604: Unable to configure certificate www.example.com:443:0 for stapling
  • [Mon Dec 04 12:40 2017] [ssl:warn] [pid 18584:tid 392] AH01906: www.example.com:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
  • [Mon Dec 04 12:40 2017] [ssl:warn] [pid 18584:tid 392] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
  • [Mon Dec 04 12:40 2017] [ssl:error] [pid 18584:tid 392] AH02217: ssl_stapling_init_cert: can’t retrieve issuer certificate! [subject: emailAddress=xxx,CN=xxx,O=xxx,L=xxx,ST=France,C=FR / issuer: emailAddress=xxx,CN=x,O=xxx,L=xxx,ST=France,C=FR / serial: xxx / notbefore: Sep 5 22:22:10 2017 GMT / notafter: Aug 12 22:22:10 2117 GMT]

#35

That’s what I meant. First you should read the documentation on the MPMs of Apache and then carefully adjust the knobs.


#36

Ok, il will see that…


#37

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.