Search papers, labs, and topics across Lattice.
This paper introduces WISE, a framework that optimizes the use of world-model imagination during the post-training of Vision-Language-Action (VLA) models by scheduling when and how imagination is invoked based on interaction relevance. By employing bounded multi-view rollouts and evaluating candidate futures with progress and completion signals, WISE effectively refines policy actions while significantly reducing computational costs. Experimental results show that WISE not only improves performance across various manipulation tasks but also enhances robustness and generalization in real-world scenarios, achieving an 80% reduction in GPU computation time compared to traditional methods.
Scheduling imagination in VLA models can cut GPU costs by 80% while boosting performance and robustness in real-world tasks.
Post-training VLA policies typically rely on supervised fine-tuning with costly expert demonstrations or reinforcement learning with expensive and potentially unstable real-world exploration. World models offer a promising alternative by evaluating candidate behaviors through imagined futures, yet effective post-training requires more than accurate prediction: imagination must be scheduled where it is useful, bounded within reliable horizons, and translated into trustworthy policy supervision. In robotic manipulation, the value of imagination varies substantially across execution stages, while extended rollouts can accumulate prediction errors and introduce unreliable learning signals. We introduce WISE (World-model-guided Imagination Scheduling for Efficient Post-training of Vision-Language-Action Models), a unified framework that coordinates when and how world-model imagination is used during policy refinement. WISE selectively invokes imagination at interaction-relevant states, performs bounded multi-view rollouts, evaluates candidate futures using progress and completion signals, and uses their relative outcomes to refine actions generated from real interaction contexts. Extensive experiments with both $\pi_0$ and $\pi_{0.5}$ demonstrate consistent improvements across diverse manipulation tasks while reducing GPU computation time by approximately 80% compared with full imagination. Real-world evaluations further show substantial gains in robustness and generalization under diverse real-world distribution shifts.
Tsinghua AI