Search papers, labs, and topics across Lattice.
This paper introduces MATCH, a novel framework that enhances the efficiency of long-context transformers by integrating in-context retrieval with sparsified attention mechanisms. By addressing the limitations of traditional attention methods, which often impose rigid constraints and degrade performance on tasks requiring long-range recall, MATCH achieves significant improvements in both synthetic and real-world natural language tasks. The results demonstrate that MATCH not only maintains the efficiency of sparse attention models but also enhances their performance through dynamic information retrieval.
MATCH reveals that integrating in-context retrieval can dramatically boost the performance of sparse attention models without sacrificing efficiency.
The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios. To improve efficiency, existing approaches often enforce rigid structural constraints such as local attention windows. However, these strategies typically lead to substantial performance degradation on tasks requiring precise long-range recall. In this work, we propose MATCH, a scalable and efficient framework that augments sparsified attention mechanisms with dynamically integrated in-context information through an efficient retrieval system. Empirical results show that MATCH significantly improves the performance of sparse-attention models on both synthetic and real-world natural-language tasks. These findings highlight the versatility of MATCH as a general approach for enhancing in-context retrieval capabilities while maintaining the efficiency benefits of sparse attention architectures.
