Search papers, labs, and topics across Lattice.
This paper investigates why improved cross-lingual alignment doesn't consistently lead to better cross-lingual transfer in encoders, despite increasing embedding similarity. Through representational analysis of XLM-R models aligned on different language pairs and fine-tuned for POS tagging and sentence classification, the authors find that embedding distances are unreliable predictors of task performance and alignment gradients are often orthogonal to task gradients. Based on these findings, they offer practical guidelines for combining cross-lingual alignment with task-specific fine-tuning.
Cross-lingual alignment can actually *hurt* transfer learning performance because aligning embeddings doesn't necessarily help with the downstream task.
Better cross-lingual alignment is often assumed to yield better cross-lingual transfer. However, explicit alignment techniques -- despite increasing embedding similarity -- frequently fail to improve token-level downstream performance. In this work, we show that this mismatch arises because alignment and downstream task objectives are largely orthogonal, and because the downstream benefits from alignment vary substantially across languages and task types. We analyze four XLM-R encoder models aligned on different language pairs and fine-tuned for either POS Tagging or Sentence Classification. Using representational analyses, including embedding distances, gradient similarities, and gradient magnitudes for both task and alignment losses, we find that: (1) embedding distances alone are unreliable predictors of improvements (or degradations) in task performance and (2) alignment and task gradients are often close to orthogonal, indicating that optimizing one objective may contribute little to optimizing the other. Taken together, our findings explain why ``better''alignment often fails to translate into ``better''cross-lingual transfer. Based on these insights, we provide practical guidelines for combining cross-lingual alignment with task-specific fine-tuning, highlighting the importance of careful loss selection.
