Search papers, labs, and topics across Lattice.
This paper introduces an inference-time multilingual steering method that leverages pretrained sparse autoencoders (SAEs) to enhance target-language features in multilingual large language models without the need for parameter updates or extensive retraining. By analyzing SAE activations across languages and selecting key layer-specific features, the authors inject these features as steering signals into the model's hidden states. The approach yields significant accuracy improvements across multiple benchmarks, with notable gains of 10.9 percentage points on XCOPA and 5.3 points on XNLI, demonstrating its effectiveness in multilingual inference tasks.
Injecting targeted language features at inference time can boost multilingual model performance by over 10 percentage points without retraining.
Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show average accuracy improvements of 10.9 percentage points on XCOPA, 5.3 points on XNLI, and 1.9 points on MGSM.
