Fix flakey vector search tests

[christophstrobl]

In slow environments the server may not be responding in time so clients run into connection timeouts.
We should add guards for test requiring more resources.