Possibly unrelated but sounds similar to https://forums.raspberrypi.com/viewtopic.php?t=362979 and that issue could be CPU bound and may be affected by key type.
See also Slow TLS handshake - #2 by _az
4 Likes