Search papers, labs, and topics across Lattice.
This study investigates the modularity of translation processes in multilingual large language models (mLLMs), revealing that translation is not only separable into conceptual and language-specific stages but also further divides into syntax and surface language production. By constructing controlled multilingual datasets and employing causal interventions and probing techniques, the authors demonstrate that mLLMs first establish target-side word order before generating the surface form of the target language. The identification of attention heads sensitive to syntactic transformations underscores the significance of syntactic structure as an independent component in translation, enhancing our understanding of mLLM functionality.
Translation in multilingual LLMs is more modular than previously thought, with syntax and surface language production operating as distinct processes.
Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.
