Search papers, labs, and topics across Lattice.
This paper introduces a Grounded World Model (GWM) for visuomotor Model Predictive Control (MPC) that operates in a vision-language-aligned latent space. GWM scores proposed actions based on the similarity between the predicted future outcome and the task instruction embedding, enabling semantic generalization. Experiments on the WISER benchmark demonstrate that GWM-MPC achieves an 87% success rate on unseen visual signals and referring expressions, significantly outperforming traditional Vision-Language-Aligned (VLA) methods.
Visuomotor control can now generalize to unseen environments and instructions by grounding world models in a vision-language latent space, outperforming standard vision-language approaches by a large margin.
In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.
