Search papers, labs, and topics across Lattice.
This paper introduces InstructMove, a benchmark designed to rigorously evaluate the instruction-following capabilities of vision-language-action (VLA) models in robot manipulation tasks. By ensuring that multiple actions are visually plausible while only one aligns with the given language instruction, InstructMove addresses the shortcomings of existing benchmarks that allow for success without true instruction grounding. Experiments demonstrate that InstructMove not only provides a controlled environment for identifying visual shortcuts but also enhances real-world manipulation performance when combined with simulation data.
Instruction-following in robot manipulation can be rigorously tested with InstructMove, revealing the true capabilities of VLA models beyond visual cues.
Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim
