Search papers, labs, and topics across Lattice.
This study investigates the cross-language reasoning capabilities of multilingual language models by analyzing their representations across six languages using the Multilingual Grade School Math dataset. Employing Centered Kernel Alignment (CKA) and a novel Geometry-Invariant Sparse Autoencoder (GI-SAE), the authors reveal that while GI-SAE enhances cross-language alignment, it does not guarantee functional interchangeability of features across languages. The findings indicate that the degree of cross-language feature sharing is model-specific and varies by architecture, highlighting the complexity of multilingual reasoning in LLMs.
Cross-language feature sharing in multilingual LLMs is model-specific and does not always translate to functional equivalence, challenging assumptions about multilingual reasoning.
Multilingual language models can solve the same mathematical problem in different languages, but it remains unclear whether they rely on shared features or on language-specific computations that only produce similar outputs. We study this question in five models from four families using the Multilingual Grade School Math (MGSM) dataset, with problems solved in English, German, French, Spanish, Russian, and Chinese, retaining problems with valid reasoning traces in all six languages and replaying those traces through the model to record representations at multiple layers. For each model, we first use Centered Kernel Alignment (CKA) to identify layers with cross-language alignment. At each selected layer, we train two sparse autoencoders (SAE): a baseline reconstruction-only model and a contrastive variant introduced in this work, the Geometry-Invariant SAE (GI-SAE). GI-SAE supplements the reconstruction loss with an Information Noise-Contrastive Estimation (InfoNCE) loss that trains the encoder to produce similar activations for traces of the same problem, regardless of language or token position. We then test whether the resulting shared features are functionally interchangeable by swapping their values between languages during the model's forward pass and measuring the resulting change in output, quantified by Kullback-Leibler (KL) divergence per feature. Although GI-SAE yields higher CKA and Jaccard similarity at nearly every layer, higher geometric similarity does not consistently imply greater functional interchangeability. We find that cross-language feature sharing is model- and architecture-dependent in this sample and appears at different depths in different models. GI-SAE primarily amplifies cross-language structure already present: the pattern is model-specific, with strengthening in Qwen, no functional benefit in Gemma, and mixed layer-dependent effects in Llama and Phi.
