Search papers, labs, and topics across Lattice.
This study evaluates the performance of three leading vector databases鈥擰drant, Milvus, and Weaviate鈥攐n high-performance computing (HPC) systems, revealing a significant scaling paradox. Despite increasing the number of distributed workers from 16 to 256, the expected improvements in query throughput were limited, with additional cores sometimes leading to a 30.67% reduction in performance. These findings underscore the incompatibility of current vector database designs with the unique demands of scientific AI workloads in HPC environments, emphasizing the need for tailored solutions.
Scaling vector databases in HPC can actually degrade performance, revealing a critical mismatch between cloud-optimized designs and scientific workloads.
Vector databases have been designed and optimized for cloud environments; however, emerging scientific AI workloads (e.g., molecular search, meteorological trajectory detection, and literature-driven hypothesis generation) demand efficient, scalable execution on HPC systems. We present a large-scale evaluation of three state-of-the-art vector databases -- Qdrant, Milvus, and Weaviate -- on two production supercomputers, scaling to 256 distributed workers across 64 compute nodes. We evaluate representative workload patterns -- mixed read/write and write-then-read -- using popular benchmarks, multimodal embeddings, and a novel real-world scientific dataset. Our results reveal that workload characteristics can limit latency reduction, additional cores can reduce query throughput by up to 30.67%, and scaling from 16 to 256 workers (16x) only yields a 5.46x improvement. This scaling paradox exposes the fundamental mismatch between cloud-oriented designs and HPC systems, highlighting the need for new, HPC-aware vector database designs.
