Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of context rot in Large Language Models (LLMs) during long-horizon tasks, revealing that as context length increases, models tend to provide uncertain or premature answers. Through extensive evaluation of four prominent models across three benchmarks, the authors identify effective strategies for mitigating context rot, including context management and post-hoc rejection sampling. Their findings indicate that combining these approaches can significantly enhance model performance while addressing the degradation caused by excessive context accumulation.
Context rot leads LLMs to falter under lengthy inputs, but targeted management and rejection strategies can restore their performance.
Extensive context has become the norm as Large Language Models (LLMs) are increasingly deployed in long-horizon tasks. The concern that increasing context length degrades model capabilities, known as context rot, has become a central issue for these applications. In this paper, we focus on deep search scenarios, aiming to investigate the rot phenomenon and its mitigation strategies. By evaluating four flagship open-source models across three benchmarks, we reveal a prevalent but unnoticed rot phenomenon: extensive context causes models to directly give up or prematurely provide uncertain answers, and this issue is exacerbated as the context grows. Through pruning experiments, we demonstrate the relationship between the accumulated context and the rot phenomenon. Furthermore, we investigate mitigating this issue through context management and post-hoc rejection sampling. For context management, we systematically evaluate seven different methods across three categories, based on performance, cost, and impact on context rot, providing clear guidance for strategy selection and usage. For rejection sampling, we develop a rot-aware filtering strategy and demonstrate its effectiveness across three aggregation methods. Finally, we show that these two approaches can be combined for further performance improvements.
