Search papers, labs, and topics across Lattice.
The paper introduces QVCache, a query-level caching system designed to improve the performance of approximate nearest neighbor (ANN) search in vector databases. QVCache leverages semantic query similarity and online learning to dynamically adjust region-specific distance thresholds, enabling recall-preserving cache hits while maintaining a bounded memory footprint. Experimental results demonstrate that QVCache, when integrated with existing ANN systems, achieves significant latency reductions (40-1000x) with sub-millisecond cache-hit latency, highlighting its potential as a crucial caching layer for scalable vector search.
Vector search gets a much-needed speed boost: QVCache slashes query latency by up to 1000x without sacrificing recall, all with a tiny memory footprint.
Vector databases have become a cornerstone of modern information retrieval, powering applications in recommendation, search, and retrieval-augmented generation (RAG) pipelines. However, scaling approximate nearest neighbor (ANN) search to high recall under strict latency SLOs remains fundamentally constrained by memory capacity and I/O bandwidth. Disk-based vector search systems suffer severe latency degradation at high accuracy, while fully in-memory solutions incur prohibitive memory costs at billion-scale. Despite the central role of caching in traditional databases, vector search lacks a general query-level caching layer capable of amortizing repeated query work. We present QVCache, the first backend-agnostic, query-level caching system for ANN search with bounded memory footprint. QVCache exploits semantic query repetition by performing similarity-aware caching rather than exact-match lookup. It dynamically learns region-specific distance thresholds using an online learning algorithm, enabling recall-preserving cache hits while bounding lookup latency and memory usage independently of dataset size. QVCache operates as a drop-in layer for existing vector databases. It maintains a megabyte-scale memory footprint and achieves sub-millisecond cache-hit latency, reducing end-to-end query latency by up to 40-1000x when integrated with existing ANN systems. For workloads exhibiting temporal-semantic locality, QVCache substantially reduces latency while preserving recall comparable to the underlying ANN backend, establishing it as a missing but essential caching layer for scalable vector search.
