Search papers, labs, and topics across Lattice.
Stellar introduces a scalable framework for multimodal document retrieval that addresses the memory overhead issues of existing methods by utilizing disk-based storage for token-level embeddings. By employing Lexical Representation-based Filtering (LRF) and Efficient Disk-backed Late Interaction (DLI), Stellar significantly reduces the candidate set and optimizes memory usage during retrieval. Experimental results show that Stellar achieves 1-2 orders of magnitude reductions in memory overhead and query latency while maintaining high retrieval effectiveness across multiple benchmarks.
Stellar slashes memory overhead and query latency by orders of magnitude, transforming multimodal document retrieval efficiency.
Multimodal document retrieval--selecting the most relevant multimodal document from a large corpus to answer a natural language query--plays an essential role in Retrieval-Augmented Generation (RAG) systems. State-of-the-art methods represent each document and query with multiple token-level embeddings and use late interaction to achieve high effectiveness. However, such multi-vector representations incur substantial memory overhead during retrieval, leading to poor scalability and hindering real-world deployment. In this paper, we present Stellar, a scalable multimodal document retrieval framework that stores token-level document embeddings on disk and loads only a small set of candidate embeddings into memory for late interaction. Stellar comprises two key components: (i) Lexical Representation-based Filtering (LRF), which fine-tunes a Multimodal Large Language Model (MLLM) as a sparse encoder to produce high-quality lexical representations, enabling efficient and effective document filtering to significantly reduce the candidate set; (ii) Efficient Disk-backed Late Interaction (DLI), which designs an on-disk token embedding storage layout guided by a balanced clustering algorithm, and dynamically loads only the necessary token embeddings into memory using a simple yet effective cost model. Extensive experiments on four real-world benchmarks and a newly presented large-scale dataset demonstrate that Stellar reduces memory overhead and query latency by 1-2 orders of magnitude compared to existing methods without compromising retrieval effectiveness.
