Search papers, labs, and topics across Lattice.
This study systematically investigates the performance of Vision-Language-Action (VLA) models when following instructions in multiple languages, revealing a significant multilingual gap where models trained predominantly on English instructions perform poorly on non-English tasks. Despite the multilingual capabilities of the underlying language models, the research shows that both instruction understanding and action execution suffer in non-English contexts. To address this issue, the authors introduce a novel fine-tuning method, Multilingual Principal Component Alignment, which effectively reduces the performance gap by aligning multilingual representations.
Multilingual instruction following in VLA models reveals a surprising performance drop, highlighting a critical gap that could hinder global applicability of these systems.
Vision-Language-Action models have recently demonstrated promising capabilities in learning generalist robot policies from large-scale multimodal data. However, most existing VLA systems are trained and evaluated primarily with English instructions, leaving their ability to understand and execute instructions in other languages largely unexplored. While the underlying large language models often possess multilingual capabilities, it remains unclear whether these multilingual capabilities transfer to VLAs during training. In this work, we present the first systematic study of multilingual instruction following in VLA models. We first construct multilingual instructions by extending existing benchmarks with translations of their instructions. Using these instructions, we evaluate several representative VLA models across a range of tasks in simulation settings. Our experiments reveal a significant multilingual gap: models trained primarily on English instructions exhibit substantial performance degradation when evaluated on other languages, even when the underlying language backbone is multilingual. We provide several findings and analyses to understand the multilingual gap. Cross-lingual transfer behavior analysis shows that performance drops correlate with both instruction understanding and action execution. Representation analyses suggest that multilingual instruction-caused representation shifts may contribute to the multilingual gap. Motivated by these findings, we further explore strategies to improve multilingual performance in VLAs. We propose a simple yet effective multilingual fine-tuning approach, Multilingual Principal Component Alignment, which leverages Principal Component Analysis to get the principal component subspace and align projected multilingual representations, effectively reducing the multilingual performance gap.
