Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates adding a low-resource language (Greek) to an open Cosmos3 vision-language-action (VLA) policy using machine-rephrased instructions, exposing severe measurement artifacts in standard embodied evaluations. The authors find that standard single-goal benchmarks produce false-positive transfer signals—scoring 84.6% on correct Greek prompts versus 82.6% under deliberately wrong instructions—and that training loss fails to predict embodied success. On a rigorous 90-task suite with adversarial controls, multilingual text towers without target-language demonstrations remain at chance floor, whereas bilingual co-training achieves 40% of English performance while halving translation-phrasing overfitting.
Standard multilingual robot benchmarks achieve an 82.6% success rate even under deliberately wrong instructions, revealing that apparent zero-shot cross-lingual transfer in VLAs is often an evaluation artifact masking total policy failure.
Robot foundation models are trained and evaluated predominantly in English, and robot demonstration corpora do not exist for most languages. We study the addition of Greek to an open vision-language-action stack using only machine-rephrased instructions and no architecture changes. The main challenge is measurement rather than translation. Several plausible instruments produce false conclusions: a color-histogram metric rewards noise, a single-goal benchmark scores 84.6% under correct Greek and 82.6% under deliberately wrong instructions, training loss fails to predict Greek success, and single-run comparisons are dominated by seed variation. On a discriminative ninety-task suite with three seeds per arm, a multilingual text tower without Greek demonstrations remains at its wrong-instruction floor, while Greek-only training exceeds its control by at most 2.7 points. Bilingual training yields a consistent 6.7-7.1 point margin over its control and reaches about two fifths of English performance. The policy also overfits the translator's phrasing; training on seven phrasings per task approximately halves this penalty. Warm-starting from a language-adapted world model and unfreezing the text tower both degrade performance. The results support two practical requirements for low-resource robot-policy localization: build a guaranteed null before trusting a metric, and replicate low-resource-language results across seeds.