Search papers, labs, and topics across Lattice.
This study investigates the cross-lingual variation in mathematical reasoning performance of large language models (LLMs) to determine whether the underlying parameters are shared across languages or language-specific. By conducting a mechanistic analysis, the authors localized and compared the model parameters that contribute to mathematical reasoning, revealing that while there is partial overlap in math-associated parameters, significant differences exist, particularly with English showing a greater number of relevant parameters compared to lower-resource languages. The findings indicate that mathematical reasoning in multilingual LLMs is characterized by a blend of shared and distinct mechanisms, highlighting the complexity of language influence on model performance.
English LLMs dominate math reasoning with a richer set of parameters, while lower-resource languages struggle with significant gaps.
Large language models (LLMs) exhibit substantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language-specific parameters or a shared mechanism that manifests differently by language. We present a cross-lingual mechanistic analysis of mathematical reasoning in LLMs, enabling us to localize and compare model parameters that support mathematical reasoning across languages. We find that the extracted math-associated parameters exhibit partial cross-lingual overlap, with the strongest overlap concentrated in intermediate model layers. We further observe that English consistently produces the largest set of math-relevant parameters, whereas lower-resource languages reveal smaller sets of relevant parameters. These results suggest that math-related behavior in multilingual LLMs is neither fully language-invariant nor fully language-specific, but instead exhibits partial cross-lingual parameter overlap with systematic language-dependent differences.
