I noticed a sudden increase in server http response time graph, and while investigating it, got here, as the only change on the server around that time was a cron job getting a new cert from let's encrypt.
I generated a cert with --preferred-chain "DST Root CA X3" and confirmed the slowdown really seems to be caused by this.
The difference is pretty significant. When running a simple wget command for a tiny static file time wget https://mydomain.com/robots.txt with the old cert I usually got response in 0.010s-0.015s, while the new cert it's in 0.050s-0.060s range.
Is this expected, or maybe there is something wrong with my config?
I'm using Debian 12 with nginx/1.25.4 and certbot 2.1.0
@Mad182, other than the chain difference, is the new cert "equal" to the previous cert?
[equal = same size and type]
If not "equal", (and if you use openssl) then you might want to check the speeds for each size and type and choose the one that works best in your system.
Cert and Privkey files are the same size, and opening them shows the same key size and algorithm, if that's what you mean, Chain is larger on the old cert.
Actually after some more investigation, it seems to only be affecting wget (and wget is used to measure the response time for the graph that got my attention).
If I run the same command substituting wget with curl, it's about the same with both certs, and loading the files in Chrome also seems to be unaffected by cert change, so it may not be such a big deal after all. No idea how to debug https connection on wget, adding --debug and --verbose doesn't say anything useful, the output is the same with both certs, minus the timestamps. But if it doesn't affect real users, not a big deal.
I got 7 identical machines and it happened on all of them at once.
They all are running up to date Debian 12 with wget, curl and certbot from official Debian repos and latest mainline nginx from nginx.org repos.
After your question I just updated the cert on one small, unrelated virtual machine running another site with a bit different setup (but also Debian 12 and nginx), and got the same result - new cert made wget much slower (but not curl).
The munin graph that got me wondering looks like this. The big jumps are from when I re-generated certs with or without "DST Root CA X3"
I checked, that the munin plugin uses wget to get the timings.
If you really want to dig down on this, I would suggest to run wget with --check-certificate=quiet for both the new and old certificate. Setting this option causes wget linked against GnuTLS (the default on Debian) to completly skip the cert verification routine (wget/src/gnutls.c at 9a35fe609c87c558153cff80fef7dea809b3cf63 · mirror/wget · GitHub).
If both certificates show the same performance with --check-certificate=quiet: The difference in performance is related to the certificate verification in GnuTLS.
If both certificates have different performance with --check-certificate=quiet: The difference in performance is most likely unrelated to the certificate and rooted in something else.
Another thing I would check is whether OCSP stapling is in the same configuration (preferably on) in both cases, as that can have a noticeable impact on verification times.
While it can be hard to parse if you are unfamiliar with lower level system details, you could use the Linux strace command to see what wget is doing under the hood and compare it to curl.
What I would personally do is run strace -ttt with both wget and curl and try to see where wget is spending a lot of time (the timestamps produced by -ttt should hopefully provide some clues); put that output to a file with the -o option. It would not surprise me that the core issue was that the GNUtls library was using a different certificate store than OpenSSL and the verification for the new root certificate was taking longer; I am aware that you tried disabling that, but perhaps that just always makes validation succeed and doesn't stop the library from chasing down the chain.
There are seven different IPs for ezgif.com, or 14, if you include ipv6 It's for load balancing. Currently 157.90.35.52 is set up with the new cert, and other servers with the old one, just for comparison.
The two chains can be swapped without changing the certificate, so I suggest trying to recreate this scenario using identical certificates and only changing the chain file. That should help figure out if this issue is caused by the chain or a minor difference in the certificate.
My quick guess is this is due to how wget is computing the trusted path, but I would bench test the matrix combination of certs vs chains to see if this can be pinpointed to the cert or chain.
Those same time command produce identical results for s1 and s2 for me. Although, you are measuring in the hundredths of seconds testing on your local net where such small differences could not reliably be measured across the public internet.
time wget https://s1.ezgif.com/robots.txt
--2024-02-15 18:39:34-- https://s1.ezgif.com/robots.txt
Resolving s1.ezgif.com (s1.ezgif.com)... 2a01:4f8:261:3e1e::2, 142.132.195.243
Connecting to s1.ezgif.com (s1.ezgif.com)|2a01:4f8:261:3e1e::2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31 [text/plain]
Saving to: ‘robots.txt.13’
robots.txt.13 100%[=======================>] 31 --.-KB/s in 0s
2024-02-15 18:39:35 (12.0 MB/s) - ‘robots.txt.13’ saved [31/31]
real 0m0.373s
user 0m0.009s
sys 0m0.004s
time wget https://s2.ezgif.com/robots.txt
--2024-02-15 18:39:41-- https://s2.ezgif.com/robots.txt
Resolving s2.ezgif.com (s2.ezgif.com)... 2a01:4f8:251:590b::2, 157.90.35.52
Connecting to s2.ezgif.com (s2.ezgif.com)|2a01:4f8:251:590b::2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31 [text/plain]
Saving to: ‘robots.txt.14’
robots.txt.14 100%[=======================>] 31 --.-KB/s in 0s
2024-02-15 18:39:42 (11.7 MB/s) - ‘robots.txt.14’ saved [31/31]
real 0m0.373s
user 0m0.003s
sys 0m0.009s
I doubt it's doing that. It's probably just the internal logic.
My quick guess is that with the long chain, the "DST" cert might be encountered quickly and lead to a fast exit from the verification routine - but with the short chain, the "X1" is towards the end. There is also a chance that wget was built against an older version of openssl and doesn't have as many optimizations and improvements, while curl and other binaries were built against a newer version that does.
It's for load balancing. Currently 157.90.35.52 is set up with the new cert, and other servers with the old one, just for comparison.
