Search papers, labs, and topics across Lattice.
This paper formalizes a fundamental trade-off in long-context sequence modeling, proving that no architecture can simultaneously achieve efficiency, compactness, and recall capabilities that scale with sequence length. The authors use information-theoretic bounds (Data Processing Inequality and Fano's Inequality) within an Online Sequence Processor framework to show that efficient and compact models can only recall O(poly(d)/log V) key-value pairs. Empirical validation across five representative architectures confirms the theoretical impossibility triangle, demonstrating that no architecture escapes the predicted trade-offs.
Long-context models face a provable "impossibility triangle": you can't have efficiency, compactness, and unbounded recall *at the same time*.
We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficiency), (ii) state size independent of sequence length (Compactness), and (iii) the ability to recall a number of historical facts proportional to sequence length (Recall). We formalize this trade-off within an Online Sequence Processor abstraction that unifies Transformers, state space models, linear recurrent networks, and their hybrids. Using the Data Processing Inequality and Fano's Inequality, we prove that any model satisfying Efficiency and Compactness can recall at most O(poly(d)/log V) key-value pairs from a sequence of arbitrary length, where d is the model dimension and V is the vocabulary size. We classify 52 architectures published before March 2026 into the triangle, showing that each achieves at most two of the three properties and that hybrid architectures trace continuous trajectories in the interior. Experiments on synthetic associative recall tasks with five representative architectures validate the theoretical bound: empirical recall capacity lies strictly below the information-theoretic limit, and no architecture escapes the triangle.
