Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces REIGN, a bi-encoder model that utilizes contextualized chunk embeddings from a frozen Guidance Network (GN) to enhance the efficiency of document retrieval over long sequences. By decoupling token-level processing from document-level reasoning and caching GN embeddings, REIGN significantly reduces training costs while maintaining competitive performance against larger models. The results show that REIGN can match or exceed the retrieval capabilities of dense long-context models with substantially fewer parameters across various benchmarks, including Wikipedia and real-world patent retrieval tasks.
REIGN achieves state-of-the-art long-document retrieval efficiency while slashing training costs by four orders of magnitude compared to traditional methods.
Dense retrieval over long documents is expensive. Token-level encoders scale quadratically in sequence length, and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs, primarily for document-to-document retrieval; single-chunk inputs stay with the GN. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets in each regime. A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.