Search papers, labs, and topics across Lattice.
This paper conducts a thorough empirical evaluation of seven leading vector database systems鈥擣AISS, Qdrant, Milvus, Weaviate, Chroma, pgvector, and LanceDB鈥攁cross diverse datasets and metrics related to retrieval quality, query performance, and resource utilization. The study reveals that while FAISS excels in throughput, Weaviate leads in recall, Qdrant offers the best latency, and LanceDB prioritizes faster index construction at the expense of retrieval quality. By providing comprehensive benchmarks and system-selection guidelines, the authors aim to fill a critical gap in the understanding of vector database performance for AI applications.
Weaviate achieves over 99% recall, setting a new standard for out-of-the-box performance in vector databases.
Vector databases have emerged as critical infrastructure for modern artificial intelligence applications, particularly retrieval-augmented generation (RAG), semantic search, and recommendation systems. Despite their growing importance, there remains a significant gap in comprehensive, reproducible benchmarks that jointly evaluate retrieval quality, query latency, throughput, and resource utilization. We present a systematic empirical evaluation of seven prominent vector database systems: FAISS, Qdrant, Milvus, Weaviate, Chroma, pgvector, and LanceDB. Our methodology spans six diverse datasets, from classical computer-vision descriptors (SIFT, GIST) to transformer-based text embeddings (MS MARCO, GloVe), encompassing over 4 million vectors at dimensionalities from 96 to 960. We measure 15 metrics spanning retrieval quality (Recall@K, Precision@K, MRR, NDCG@K, Hit Rate@K), query performance (latency percentiles, QPS, cold-start latency), and resource consumption (index build time, memory, storage). On SIFT1M, FAISS achieves the highest single-node throughput (866 QPS) but lacks database operational features; Weaviate provides the best out-of-the-box recall (>99%); Qdrant offers the best latency among full databases (4.55~ms median); and LanceDB trades retrieval quality for substantially faster index construction. We derive system-selection guidelines for practitioners and release our benchmarking framework as open-source software.
