Search papers, labs, and topics across Lattice.
The paper addresses the brittleness of Vision-Language-Action models (VLAs) to environmental shifts by proposing Perturbation learning with Delayed Feedback (PDF), a test-time adaptation framework that avoids fine-tuning. PDF uses uncertainty-based data augmentation and action voting to mitigate spurious correlations, and learns a perturbation module to adjust action logits based on delayed feedback. Experiments on LIBERO and Atari demonstrate significant performance gains over vanilla VLA and VLA with other test-time adaptation methods, showing the effectiveness of PDF in improving the robustness of VLAs.
VLAs can learn to adapt to new environments at test time without any fine-tuning, achieving significant performance gains on robotic manipulation and Atari games.
Vision-Language-Action models (VLAs) achieve remarkable performance in sequential decision-making but remain fragile to subtle environmental shifts, such as small changes in object pose. We attribute this brittleness to trajectory overfitting, where VLAs over-attend to the spurious correlation between actions and entities, then reproduce memorized action patterns. We propose Perturbation learning with Delayed Feedback (PDF), a verifier-free test-time adaptation framework that improves decision performance without fine-tuning the base model. PDF mitigates the spurious correlation through uncertainty-based data augmentation and action voting, while an adaptive scheduler allocates augmentation budgets to balance performance and efficiency. To further improve stability, PDF learns a lightweight perturbation module that retrospectively adjusts action logits guided by delayed feedback, correcting overconfidence issue. Experiments on LIBERO (+7.4\% success rate) and Atari (+10.3 human normalized score) demonstrate consistent gains of PDF in task success over vanilla VLA and VLA with test-time adaptation, establishing a practical path toward reliable test-time adaptation in multimodal decision-making agents. The code is available at \href{https://github.com/zhoujiahuan1991/CVPR2026-PDF}{https://github.com/zhoujiahuan1991/CVPR2026-PDF}.
Tsinghua AI