Search papers, labs, and topics across Lattice.
This paper introduces FeelWorld, a hierarchical visuo-tactile world model that enhances contact prediction and planning by jointly predicting visual latents and tactile states, specifically contact and slip states. The model employs a contact-gated asymmetric attention mechanism to differentiate between visual-only predictions in free space and joint visuo-tactile predictions during contact, thereby improving the accuracy of imagined physical interactions. Experimental results demonstrate that FeelWorld significantly reduces prediction error and achieves an impressive 81.7% zero-shot planning success rate in tasks involving complex physical interactions like grasping and insertion.
FeelWorld achieves a 61% reduction in prediction error compared to visual-only baselines, revolutionizing how we model tactile interactions in robotics.
Humans plan physical interactions by imagining the possible outcomes of candidate actions. However, existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing imagined futures that appear visually plausible but violate physical dynamics. We introduce FeelWorld, a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. FeelWorld organizes these states hierarchically as contact state, a 3D tactile latent that encodes force-related information, and slip state. These states are jointly predicted by a shared latent dynamics model with explicit supervision. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, we introduce a contact-gated asymmetric attention mechanism that maintains a visual-only prediction pathway before contact and enables joint visuo-tactile dynamics prediction during contact. The model is further trained with autoregressive rollouts and context noise injection to improve robustness to compounding errors. The predicted contact and slip states also support contact-aware CEM planning. Experiments on chip grasping, fruit grasping, and USB insertion show that FeelWorld reduces 10-step LPIPS from 0.084 to 0.058 and maintains an LPIPS that is 61% lower than that of the visual baseline after an 80-step autoregressive rollout. FeelWorld also achieves an average zero-shot planning success rate of 81.7%, providing an effective approach for incorporating tactile sensing into world models.
