Filesize seems to have no effect. I also tried 20kb css file and 200kb image, there was no difference.
But what is interesting - if I run the requests from my home computer (also debian 12), which is 1000+km away and adds pretty stable ~200ms of latency, I don't see any difference in timings, even after running a bunch of times.
Running the request both from the machine itself, and from other machines in the datacenter, there is the ~50ms difference pretty much always.
So I guess it only manifests if the response is very fast.