Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper investigates the ability of Vision-Language-Action (VLA) models to follow "How-level" instructions, which specify execution methods rather than just task goals. Through diagnostic experiments and the introduction of the SIMPLER-IN benchmark, the study reveals that current VLAs do not effectively leverage How-level instructions, with their inclusion sometimes even degrading performance. The findings suggest that current training paradigms, pairing single episodes with single instructions, are insufficient for robust How-level instruction following, highlighting it as an open research challenge.
Current VLA models fail to benefit from "How-level" instructions, with enriched training data even decreasing task success.
Vision-language-action (VLA) models are expected to follow instructions like their language backbones, yet prior work has focused on What-level instructions that specify the task rather than How-level instructions that specify how to carry it out. Through our diagnoses, we find that How-level instructions do not help current VLAs and lower task success. We hypothesize that pairing each training episode with a single instruction limits How-level instruction following, and show that training with segment-level enriched instructions makes actions respond to such instructions without consistently raising task success. To measure How-level instruction following beyond standard benchmarks, we introduce SIMPLER-IN, where success requires conditions given only in the instruction. It shows that the gained responsiveness does not extend to demanding or out-of-distribution How-level instructions. How-level instruction following thus remains an open challenge.