Search papers, labs, and topics across Lattice.
This paper introduces D-NOVA, a hardware-software co-designed in-storage retrieval accelerator that addresses the latency and energy overhead associated with dense vector retrieval in Retrieval-Augmented Generation (RAG) systems. By embedding search functionality directly into the NAND memory array using a novel distance metric called Dual-Bound Tight Similarity Sensing (DTS), D-NOVA significantly reduces reliance on external processors and mitigates bandwidth limitations. The results show that D-NOVA achieves up to 41.7x faster performance and 71x greater energy efficiency compared to traditional CPU baselines, while also outperforming existing in-storage RAG accelerators in throughput and energy efficiency.
D-NOVA accelerates in-storage retrieval by over 41x, revolutionizing the efficiency of RAG systems and challenging the need for external processing.
Retrieval-Augmented Generation (RAG) enhances the factual grounding of large language model (LLM) inference by retrieving relevant information from external knowledge bases. However, its dense vector retrieval introduces significant latency and energy overhead, becoming the primary performance bottleneck. Although recent in-storage accelerators aim to reduce data movement, they still rely on host or embedded processors outside the memory, where nearly 70% of the total retrieval time is spent. As a result, they cannot fully overcome the bandwidth limitations, leading to yet another memory bottleneck. To tackle these limitations, we present D-NOVA, a hardware-software co-designed in-storage retrieval accelerator. D-NOVA executes an inverted file (IVF)-based hierarchical retrieval pipeline by deeply embedding the search functionality directly into the NAND memory array. This is achieved by incorporating a new distance metric, Dual-Bound Tight Similarity Sensing (DTS), which is specifically tailored for searching within the NAND string. In addition, we introduce a lightweight contrastive adapter that maps embedding vectors into a DTS-friendly domain, recovering near-software recall while improving performance and energy efficiency. D-NOVA is up to 41.7x faster and 71x more energy-efficient than a CPU baseline, and achieves 12.13x higher throughput while being up to 1.26x more energy-efficient than state-of-the-art in-storage RAG accelerators, demonstrating the potential of fully in-storage vector search for scalable RAG acceleration.
