Search papers, labs, and topics across Lattice.
This paper addresses the performance degradation of vision-language-action (VLA) models when transferred to new robotic embodiments by introducing a self-supervised finetuning method that generates online interaction rollouts from a zero-shot VLA. The approach effectively retains the model's original instruction-following capabilities while enhancing its ability to learn new tasks from expert data, resulting in improved sample efficiency. Experiments on both a real ALOHA robot and a simulation benchmark demonstrate that the proposed method yields robust multi-task policies that generalize well across various tasks.
Self-supervised finetuning can preserve a VLA model's instruction-following abilities while significantly boosting its performance on new robotic tasks.
State-of-the-art vision-language-action (VLA) models such as $\pi_{0.5}$ exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new robots, even minor mismatches in hardware configuration relative to pretraining can cause severe performance drops. Finetuning the VLA on in-domain expert data from the new embodiment improves performance on the expert task but leads to a loss in its original instruction following and behavioral priors. In this paper, we propose a self-supervised method that generates online interaction rollouts from the zero-shot VLA as additional training data for finetuning. Our experiments show this finetuning scheme yields strong multi-task policies that, on the target robot, (1) inherit prior tasks distilled from the zero-shot model, (2) enable generalist instruction following, while (3) learning new skills from expert data with improved sample efficiency. We demonstrate the success of our approach across test sets probing generalization on a real ALOHA robot and a new simulation benchmark in RoboTwin. Video results are available at https://self-supervised-control.pages.dev/
