Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the impact of lexical density on the effective context window of large language models (LLMs), revealing that high rates of distinct information introduction significantly degrade performance. Using three controlled benchmarks with consistent length but varying information density, the authors demonstrate that LLMs experience a drastic drop in retrieval accuracy in high-density contexts, falling below 60% despite near-perfect performance in sparser environments. The findings highlight that effective context capacity is not solely determined by input length or information position, but also critically influenced by lexical density, which has significant implications for the deployment of LLMs in real-world scenarios with compact, information-rich inputs.
High lexical density can cause LLMs to lose over 40% of their retrieval accuracy, challenging assumptions about context limits in AI applications.
Input length and the position of relevant information are widely cited as the primary causes of degraded LLM long-context performance. Here, we study lexical density -- the rate at which a context introduces distinct information -- as a third, largely overlooked factor that systematically reduces the effective context window of LLMs. We quantify the impact of lexical density on open-weight LLMs (9B-685B) using three "find-the-needle" style benchmarks with identical length (~12k tokens) and controlled needle position, but increasing density of information. We observe a sharp performance collapse in higher-density benchmarks: models that are near-perfect in sparse contexts drop below 60% retrieval score on denser ones. To rule out task-type confounds, we vary and control the density within each benchmark while keeping all other properties unchanged. Reducing density generally restores performance, especially in the high-density regimes where degradation appears. These results show that effective context capacity is a function of lexical density, with direct implications for real-world LLM systems operating on compact, information-rich inputs.