Search papers, labs, and topics across Lattice.
Orthros attention addresses the quadratic complexity bottleneck in long-context Transformers by introducing a phase-aware decoupling mechanism that dynamically switches between linear complexity prefilling and full attention decoding. This unified module achieves lower asymptotic complexity in prevalent long-context scenarios, outperforming vanilla Transformers in empirical evaluations across language modeling, conversion feasibility, and inference speed. Orthros demonstrates significant prefilling speedups and preserves long-context recall, positioning it as a strong candidate for industrial applications.
A single attention module dynamically switches between linear prefilling and full decoding, achieving $21.9\times$ faster prefilling and Transformer-level long-context recall without the quadratic bottleneck.
Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling linear complexity prefilling and full attention decoding. Our design grants Orthros attention a lower asymptotic complexity compared to vanilla Transformers in prevalent long-context scenarios characterized by heavy prefilling and light decoding. Beyond theoretical formulation and complexity analysis, extensive empirical evaluations demonstrate Orthros's superiority across multiple dimensions. First, we validate its fundamental language modeling capabilities, demonstrating scalability across varying model sizes and strong competitiveness in comprehensive architectural comparisons. Second, Orthros exhibits exceptional conversion feasibility via a data-efficient Transformer-to-Orthros conversion, yielding performance that even surpasses the original Transformer. Finally, Orthros achieves a $21.9\times$ prefilling speedup over Transformers with end-to-end inference gains, while preserving long-context capabilities, as demonstrated by near-perfect 128K needle-in-a-haystack results and Transformer-level performance on real-world in-context recall tasks. Overall, these advantages in high performance, flexibility, and efficiency position Orthros as a promising candidate to succeed the Transformer in industrial applications.
