Search papers, labs, and topics across Lattice.
This paper investigates the multilingual reasoning gap in LLMs, where models perform worse when reasoning in non-English languages compared to English. They fine-tuned Qwen3-8B models on multilingual reasoning datasets across six languages, finding a significantly smaller native reasoning gap (1.9-3.5%) than previously reported when using comparable supervision. By analyzing weight-space alignment, they identified a language-agnostic reasoning core and introduced a Layer Swap technique, transferring mid-layers from an English specialist to native specialists, effectively closing the remaining reasoning gap.
Turns out, you can boost a multilingual LLM's reasoning in any language just by swapping in the mid-layers from an English-reasoning specialist.
Recent reasoning Large Language Models produce a chain-of-thought (CoT) predominantly in English, even when prompted in non-English languages. Prior work suggests that forcing the CoT to remain in the input language (\emph{native reasoning}) substantially degrades performance relative to allowing the model to reason in English before answering in the input language (\emph{English-pivoted reasoning}). However, most studies of this native reasoning gap rely on inference-time interventions or limited native-language training data. We revisit this comparison at a larger scale and under comparable supervision. We construct long multilingual reasoning datasets across six languages (English, French, German, Spanish, Chinese and Swahili); fine-tune specialists in both native and English-pivoted regimes on top of \texttt{Qwen/Qwen3-8B-Base}, and evaluate across mathematics, science, general knowledge, and code. In this setting, the average native reasoning gap shrinks to 1.9--3.5\% across the five non-English languages, considerably smaller than previously reported. Weight-space analysis of the native specialists reveals aligned fine-tuning updates in the middle layers and divergence in the outer layers. This points to a largely language-agnostic reasoning core surrounded by language-specific layers. Exploiting this structure, we introduce a Layer Swap: transferring the English specialist's stronger reasoning mid-layers into each native specialist, closing most of the native reasoning gap across the five non-English languages while preserving CoT in the target language. We release all models and datasets.
