Search papers, labs, and topics across Lattice.
This paper introduces miniReranker, a novel approach to multimodal reranking that leverages a vision-first formulation to enhance cache reuse and improve reranking performance. By implementing strategies such as early exit to reduce model depth, restricting cross-segment attention, and employing embedder-guided pruning for visual tokens, miniReranker achieves a remarkable reduction in reranking runtime to less than 1% of traditional dense implementations. Importantly, this efficiency gain comes with minimal performance loss, maintaining over 96% of the dense model's effectiveness.
miniReranker slashes reranking runtime to under 1% of dense implementations while retaining over 96% of performance, revolutionizing efficiency in multimodal tasks.
Multimodal large language models (MLLMs) have recently shown strong potential as point-wise rerankers by directly modeling query--document relevance through next-token prediction. However, point-wise reranking suffers from substantial repeated computation across query--document pairs, while the causal structure of transformers allows only prefix segments to be reused via pre-caching. To address the misalignment of existing query-first and document-first formats with both VQA-style prompting and computation-aware reuse, we propose a \textit{vision-first} formulation that improves both cache reuse efficiency and reranking performance. However, the remaining cost is still considerable and stems from three main sources: (1) \textit{model depth}, for which we reduce active parameters via early exit; (2) \textit{cross-segment attention}, which we restrict to a narrow interaction band across a few layers; and (3) \textit{visual tokens}, where we reduce the number of tokens via embedder-guided pruning. Together, these designs form miniReranker, which reduces reranking runtime to <1% of the dense implementation under high-reuse settings for a single query, while preserving >96% of the dense model performance.
