Search papers, labs, and topics across Lattice.
The paper investigates the role of attention heads in multilingual language models, identifying "Retrieval-Transition Heads" (RTHs) that are crucial for target-language generation and cross-lingual transfer. Through analysis of multilingual models like Qwen-2.5 and Llama-3.1, the authors demonstrate that RTHs are distinct from standard retrieval heads and play a more significant role in chain-of-thought reasoning across languages. Experiments involving masking RTHs on benchmarks like MMLU-ProX and MGSM reveal a greater performance degradation compared to masking retrieval heads, highlighting their importance for cross-lingual tasks.
Multilingual LLMs have special attention heads that are more important for reasoning than standard retrieval heads, and selectively masking them tanks performance.
Recent work has identified a subset of attention heads in Transformer as retrieval heads, which are responsible for retrieving information from the context. In this work, we first investigate retrieval heads in multilingual contexts. In multilingual language models, we find that retrieval heads are often shared across multiple languages. Expanding the study to cross-lingual setting, we identify Retrieval-Transition heads(RTH), which govern the transition to specific target-language output. Our experiments reveal that RTHs are distinct from retrieval heads and more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuaD) and two model families (Qwen-2.5 and Llama-3.1), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Our work advances understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.
