Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
Vision-language-action (VLA) models exhibit a critical failure mode, termed instruction-action binding, where they respond to language and vision but fail to integrate them for correct action selection under counterfactual scenarios. This failure stems from the model's tendency to cue familiar trajectory families and adjust execution, leading to retained source behaviors or switches to other demonstrated tasks rather than the required action. Equivariant Counterfactual Training (ECT) addresses this by augmenting data with counterfactually distinct demonstrations and employing a loss function that trains corresponding demonstrations jointly, significantly improving generalization on unseen tasks and real-world robotic manipulation.
VLA models fail not by ignoring instructions, but by misbinding them to incorrect action trajectories, a problem solved by training with counterfactually paired demonstrations.
Vision-language-action (VLA) models can exceed 90% success on in-distribution tasks and withstand nuisance changes that preserve the required action, yet fail under counterfactual changes that demand a different action. Aggregate robustness scores can therefore conceal a more specific failure, in which a policy responds to both language and vision yet does not combine them to select the action the task requires. We call this failure instruction-action binding. Instructions cue familiar trajectory families, and visual feedback adjusts their execution. Behavioral analyses of fine-tuned $π_{0.5}$ and GR00T-N1.7 policies reveal that failed rollouts often retain the source behavior or switch to another demonstrated task. These switches show that language is not simply ignored. Readouts and interventions connect these choices to task-conditioned internal states. Our analysis of the imitation objective shows how narrow conditional action support can leave grounded and instruction-keyed solutions indistinguishable on the demonstrations. This motivates Equivariant Counterfactual Training (ECT), which acts at two levels. ECT data supply valid demonstrations in which the same instruction requires different actions in distinguishable scenes, while the ECT loss trains each demonstration with its counterpart in the same update. In a controlled LIBERO-PRO comparison, full ECT raises $π_{0.5}$'s mean position-swap success from 36% to 59%. On CALVIN, where counterparts already occur in the original data, the ECT loss improves five-task completion without new demonstrations. On a real UR5e under a fixed demonstration budget, full ECT raises unseen-position success from 8% to 88%.