Search papers, labs, and topics across Lattice.
This study addresses the inefficiencies of causal self-attention in transformers by analyzing the impact of state update designs in a frozen-backbone framework. The authors reveal that softmax operations depend on key-specific, rank-1 orthogonal projections, leading to the conclusion that delta-style networks outperform traditional gated accumulation methods. By implementing structural interventions like sink tokens and fixed-budget cache routing, they achieve significant improvements in linearization across large models, surpassing previous baselines on MMLU while maintaining long-context retrieval capabilities.
Delta-style networks outperform traditional methods by leveraging key-dependent projections, revealing a new path to efficient transformer inference.
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.
