Well, this depends on your definition of "slow". The TLS handshake does add overhead to a connection that would otherwise not be there. There are a bunch of relatively computationally expensive operations to perform, so a TLS handshake does usually take a few milliseconds. The exact "how much" cannot be easily quantified, as this depends on many variables, including network performance/latency, computer performance and so on.
As an example, I benchmarked some TLS handshakes times to various nationwide servers with Firefox and they were commonly scattered around ~50ms, sometimes less. Considering my average network latency of ~20ms, that's a bit more than 2 Round-Trip-Times (RTT) of latency. The vast majority of this delay comes from network latency, not from the computations.
For comparison, that's about double of the setup time of a typical TCP connection (which is around 1 RTT until the client can send data) and is also a typical DNS latency for some ISPs.
Adding this all together, DNS lookup, TCP setup (TCP handshake), TLS setup (TLS handshake) can add up to over a quarter of a second in suboptimal conditions. And that's just the connection setup: Next step is the actual HTTP request, and even fast HTTP servers often have "Time to first byte" (TTFB) times of over a hundred milliseconds - this actually the major delay part for some initial requests. Considering all of this, the TLS handshake usually only plays a minor (or average) role in overal latency.
Of course much of this is highly optimized: DNS lookups are cached and the TCP/TLS connections are usually "kept alive" - that is, for subsequent requests they're kept open, such as to avoid having to negotiate TLS for every single HTTP request/response. Instead they're send over the same connection so that the delay only applies for the initial request, but not for subsequent ones.
In addtion to the above, TLS also has a feature called session resumption (commonly enabled on TLS servers, not supported by all TLS clients). This allows a TLS client to resume an older TLS session, even if the underlying TCP connection was terminated previously. This allows to skip major parts of the TLS handshake and reduces TLS setup times greatly.
Now, your setup might be different - if your WebDav server for example aggressively shuts down lingering connections, this may hurt performance. Session resumption may also not work, depending on what clients are connecting and how the server is configured. You might also have different computer performance or network latencies, where the TLS handshake is more significant than in the average case - we don't know. But if I had to guess, I would presume the issue is more in the HTTP TTFB, which has nothing to do with the TLS handshake - but that is where server's are commonly slow.
