Search papers, labs, and topics across Lattice.
The paper introduces RoboSemanticBench (RSB), a benchmark designed to evaluate the semantic grounding capabilities of vision-language-action (VLA) models in action prediction tasks. By assessing how well these models can use complex instruction semantics to select and manipulate physical targets, the study reveals that many VLA models struggle to select the semantically correct answer, often performing at near-random rates despite successful grasping. This highlights a significant disconnect between the semantic understanding provided by pretrained models and their practical application in robotic action prediction.
Many VLA models fail to translate semantic understanding into accurate action selection, often performing at near-random levels despite successful grasping.
Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB), an embodied benchmark for diagnosing semantic grounding in action prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction.
