Search papers, labs, and topics across Lattice.
This paper addresses the issue of instruction blindness in vision-language-action (VLA) models, where finetuning on limited robot data often leads to models that prioritize visual cues over language instructions. By employing flatness-preserving optimization techniques, specifically sharpness-aware minimization during finetuning, the authors achieve a remarkable improvement of over 60% in instruction following across various benchmarks without requiring additional data or architectural changes. This work highlights the importance of loss landscape characteristics in enhancing model robustness and adherence to language instructions in VLA systems.
Instruction blindness in VLA models can be mitigated by optimizing for flatter loss landscapes, leading to over 60% better adherence to language instructions.
Vision-language-action (VLA) models have the potential for open-world generalization by leveraging pretrained vision-language representations, yet downstream finetuning on limited robot data often degrades these representations, leading to brittle policies that ignore language instructions in favor of visual shortcuts, a failure mode we term instruction blindness. We hypothesize that standard finetuning with limited data applies gradients to a sparse set of points, which manifests as a sharp loss landscape with high-curvature minima. We propose to address this directly through flatness-preserving optimization while finetuning on the exact same data, where learning a flatter landscape results in a model more robust to perturbations in the weight space. Specifically, we demonstrate that simply applying sharpness-aware minimization during VLA finetuning significantly improves instruction following by over 60% across multiple simulation and real-world benchmarks without additional data, architectural modification, or retraining. We further analyze the effect of selective sharpness, quantify its effects, and show that our approach is complementary to existing guidance techniques. Project page can be found at https://haochenz11.github.io/papers/flatness-vla/.
CMU ML