Pageload Times Increased after DST Root X1 / Shortening Change


I noticed a sudden increase in server http response time graph, and while investigating it, got here, as the only change on the server around that time was a cron job getting a new cert from let's encrypt.
I generated a cert with --preferred-chain "DST Root CA X3" and confirmed the slowdown really seems to be caused by this.

The difference is pretty significant. When running a simple wget command for a tiny static file time wget with the old cert I usually got response in 0.010s-0.015s, while the new cert it's in 0.050s-0.060s range.

Is this expected, or maybe there is something wrong with my config?

I'm using Debian 12 with nginx/1.25.4 and certbot 2.1.0

@Mad182, other than the chain difference, is the new cert "equal" to the previous cert?
[equal = same size and type]

If not "equal", (and if you use openssl) then you might want to check the speeds for each size and type and choose the one that works best in your system.

openssl speed


Cert and Privkey files are the same size, and opening them shows the same key size and algorithm, if that's what you mean, Chain is larger on the old cert.

Actually after some more investigation, it seems to only be affecting wget (and wget is used to measure the response time for the graph that got my attention).

If I run the same command substituting wget with curl, it's about the same with both certs, and loading the files in Chrome also seems to be unaffected by cert change, so it may not be such a big deal after all. No idea how to debug https connection on wget, adding --debug and --verbose doesn't say anything useful, the output is the same with both certs, minus the timestamps. But if it doesn't affect real users, not a big deal.


That's… weird. Is it reproducible on multiple client systems? Or is it just one weird system that has wget and curl giving different timings?


I got 7 identical machines and it happened on all of them at once.
They all are running up to date Debian 12 with wget, curl and certbot from official Debian repos and latest mainline nginx from repos.

After your question I just updated the cert on one small, unrelated virtual machine running another site with a bit different setup (but also Debian 12 and nginx), and got the same result - new cert made wget much slower (but not curl).

1 Like

The munin graph that got me wondering looks like this. The big jumps are from when I re-generated certs with or without "DST Root CA X3"
I checked, that the munin plugin uses wget to get the timings.

If you really want to dig down on this, I would suggest to run wget with --check-certificate=quiet for both the new and old certificate. Setting this option causes wget linked against GnuTLS (the default on Debian) to completly skip the cert verification routine (wget/src/gnutls.c at 9a35fe609c87c558153cff80fef7dea809b3cf63 · mirror/wget · GitHub).

  • If both certificates show the same performance with --check-certificate=quiet: The difference in performance is related to the certificate verification in GnuTLS.
  • If both certificates have different performance with --check-certificate=quiet: The difference in performance is most likely unrelated to the certificate and rooted in something else.

Another thing I would check is whether OCSP stapling is in the same configuration (preferably on) in both cases, as that can have a noticeable impact on verification times.


Adding --check-certificate=quiet has no effect, still slow with the new cert.

My nginx config file has

ssl_stapling on;
ssl_stapling_verify on;

I made no changes in the nginx config, just executed certbot renew command and then reloaded nginx.


Can you verify that stapling actually works on the new cert?


While it can be hard to parse if you are unfamiliar with lower level system details, you could use the Linux strace command to see what wget is doing under the hood and compare it to curl.

What I would personally do is run strace -ttt with both wget and curl and try to see where wget is spending a lot of time (the timestamps produced by -ttt should hopefully provide some clues); put that output to a file with the -o option. It would not surprise me that the core issue was that the GNUtls library was using a different certificate store than OpenSSL and the verification for the new root certificate was taking longer; I am aware that you tried disabling that, but perhaps that just always makes validation succeed and doesn't stop the library from chasing down the chain.


ssllabs shows I have OCSP stapling onabled with both versions of the cert.

New shortened chain cert:

Old cert:

1 Like

Why are there two different IPs?


There are seven different IPs for, or 14, if you include ipv6 :slight_smile: It's for load balancing. Currently is set up with the new cert, and other servers with the old one, just for comparison.


Are you testing by using the domain name in one case and the IP address in the other?

Because if I poke that IP address I get 301 redirected to the domain name. That doesn't happen if I use the domain name directly.

And, curl just returns the 301 result whereas wget follows it


The servers themselves has domain name in the /etc/hosts pointing to, so it's not a dns issue.

Externally you can test it with this url: - old cert - new cert

This result is pretty constant, curl gives the same time (it was always slightly slower than wget) but wget takes 5x more time with the new cert.


The two chains can be swapped without changing the certificate, so I suggest trying to recreate this scenario using identical certificates and only changing the chain file. That should help figure out if this issue is caused by the chain or a minor difference in the certificate.

My quick guess is this is due to how wget is computing the trusted path, but I would bench test the matrix combination of certs vs chains to see if this can be pinpointed to the cert or chain.


wget seems to "take longer" with one than the other - why?
It's having to lookup something outside its' locally trusted root store?


Those same time command produce identical results for s1 and s2 for me. Although, you are measuring in the hundredths of seconds testing on your local net where such small differences could not reliably be measured across the public internet.

time wget
--2024-02-15 18:39:34--
Resolving ( 2a01:4f8:261:3e1e::2,
Connecting to (|2a01:4f8:261:3e1e::2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31 [text/plain]
Saving to: ‘robots.txt.13’

robots.txt.13         100%[=======================>]      31  --.-KB/s    in 0s

2024-02-15 18:39:35 (12.0 MB/s) - ‘robots.txt.13’ saved [31/31]

real    0m0.373s
user    0m0.009s
sys     0m0.004s

time wget
--2024-02-15 18:39:41--
Resolving ( 2a01:4f8:251:590b::2,
Connecting to (|2a01:4f8:251:590b::2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 31 [text/plain]
Saving to: ‘robots.txt.14’

robots.txt.14         100%[=======================>]      31  --.-KB/s    in 0s

2024-02-15 18:39:42 (11.7 MB/s) - ‘robots.txt.14’ saved [31/31]

real    0m0.373s
user    0m0.003s
sys     0m0.009s

I doubt it's doing that. It's probably just the internal logic.

My quick guess is that with the long chain, the "DST" cert might be encountered quickly and lead to a fast exit from the verification routine - but with the short chain, the "X1" is towards the end. There is also a chance that wget was built against an older version of openssl and doesn't have as many optimizations and improvements, while curl and other binaries were built against a newer version that does.

1 Like

What OS are you testing on? Is this wget behavior seeming to be specific to Debian 12?