Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates the phenomenon of "token overflow" in retrieval-augmented generation (RAG) systems employing soft compression, where compressed token representations lose task-relevant information. They propose a methodology to characterize and detect token overflow, exploring both query-agnostic saturation statistics and query-aware probing classifiers. The study finds that lightweight probing classifiers leveraging both query and context representations achieve an average AUC-ROC of 0.72 in detecting overflow across HotpotQA, SQuADv2, and TriviaQA datasets, outperforming query-agnostic methods.
Query-aware probes can detect when compressed token representations in RAG systems have lost too much information, opening the door to pre-LLM gating strategies that avoid compression-induced errors.
Efficient long-context processing remains a crucial challenge for contemporary large language models (LLMs), especially in resource-constrained environments. Soft compression architectures promise to extend effective context length by replacing long token sequences with smaller sets of learned compressed tokens. Yet, the limits of compressibility -- and when compression begins to erase task-relevant content -- remain underexplored. In this paper, we define token overflow as a regime in which compressed representations no longer contain sufficient information to answer a given query, and propose a methodology to characterize and detect it. In the xRAG soft-compression setting, we find that query-agnostic saturation statistics reliably separate compressed from uncompressed token representations, providing a practical tool for identifying compressed tokens but showing limited overflow detection capability. Lightweight probing classifiers over both query and context xRAG representations detect overflow with 0.72 AUC-ROC on average on HotpotQA, SQuADv2, and TriviaQA datasets, demonstrating that incorporating query information improves detection performance. These results advance from query-independent diagnostics to query-aware detectors, enabling low-cost pre-LLM gating to mitigate compression-induced errors.