Search papers, labs, and topics across Lattice.
This paper introduces AHEAD, a novel predictive framework that enhances Vision-Language-Action (VLA) models by incorporating a motion-aware latent world model to address the challenges posed by dynamic object manipulation. By forecasting future object states based on optical flow and employing a saliency mask to focus on relevant patches, AHEAD significantly improves the performance of VLA models in dynamic environments, achieving a success rate of 79 to 97% across various simulation scenarios. In physical tests, AHEAD outperformed all baselines, achieving near-perfect success rates in complex manipulation tasks, demonstrating its effectiveness in real-world applications.
AHEAD transforms static VLA models into dynamic manipulators, achieving up to 97% success in scenarios where traditional methods fail spectacularly.
Vision-Language-Action (VLA) models generalize across static manipulation but fail when objects move during task execution. They map the current observation to an action and assume the scene is stationary between observation and execution, so at any non-trivial object speed the resulting latency exceeds the time available to grasp. We close this gap with AHEAD (Anticipatory Horizon Extrapolation with Adaptive Dynamics), a predict-then-act wrapper that augments a frozen VLA with a motion-aware latent world model. A small world model trained on manipulation video forecasts future patch tokens in the VLA's feature space, conditioned on per-token velocity and acceleration from optical flow. A language-and-motion saliency mask concentrates prediction on task-relevant patches, and the model rolls forward for an adaptive horizon, halting when prediction uncertainty crosses a threshold. The frozen action decoder then receives the predicted future tokens in place of the current ones. AHEAD adds 4.9M parameters to a frozen 7B OpenVLA and reaches 79 to 97% success across 20 dynamic simulation scenarios where the strongest baseline reaches 31 to 58%. On a physical UFactory xArm 7, AHEAD succeeds on 29/30 to 30/30 on three conveyor and rolling-ball tasks, 23/30 on paddle interception, and 19/30 on projectile catching where every baseline scores 0/30.
CMU ML