Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The authors investigate how hybrid attention architectures—which combine recurrent states and full softmax attention to scale context lengths—impact multilingual representation learning. Internal interpretability probing reveals that cross-lingual representation alignment does not develop smoothly, but instead exhibits a sharp spike at the first full-attention layer encountered in the network. Exploiting this phenomenon, alternative layer configurations that place full attention earlier in the model consistently outperform conventional interleaving heuristics, training up to 2.5X faster on multilingual distillation tasks.
Standard hybrid architectures order their layers backwards for multilingual data: placing a full-attention layer first triggers immediate cross-lingual representation alignment and accelerates distillation by up to 2.5X.
In response to the growing demand for long sequences in agentic and reasoning use cases, many state-of-the-art LLMs combine multiple variants of attention to mitigate the quadratic complexity of traditional softmax attention. These hybrid attention LLMs aim to balance the strengths and limitations of full attention and alternatives based on recurrence. This work presents a first study of how hybrid attention impacts the multilinguality of LLMs. Beyond the impact on long sequences in poorly tokenized languages, our study is motivated by the possibility that the inductive biases of the recurrent state alter linguistic processing. Our interpretability analysis confirms this, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-attention layers. Across diverse models, we notably observe a pronounced spike in cross-lingual alignment around the first full-attention layer. These findings lead us to question the conventional ordering of attention layers. In distillation experiments on multilingual data, all alternative layer orderings outperform the standard throughout training, learning up to 2.5X faster. These stark, replicable results prompt our theory that multilingual models would benefit from starting with a full-attention layer rather than recurrent layers.