Search papers, labs, and topics across Lattice.
This paper introduces VT-WAM, a Visual-Tactile World Action Model designed to enhance contact-rich manipulation by jointly modeling future visual predictions, tactile deformation dynamics, and action predictions. By employing Asymmetric Mixture-of-Transformers (MoT) attention and contact-gated Action-Visual-Tactile Attention Guidance (AVTAG), VT-WAM effectively integrates visual and tactile information, leading to significant improvements in action performance. The model achieves a 71.67% average success rate across six manipulation tasks, surpassing previous benchmarks by substantial margins, underscoring the importance of tactile dynamics in action generation.
Tactile dynamics are crucial for contact-rich manipulation, and VT-WAM outperforms existing models by 26.67% to 35.84% by effectively integrating visual and tactile cues.
Contact-rich manipulation requires policies to react to local deformation, pressure, slip, and friction, yet these cues are temporally sparse and often invisible in visual observations. Existing visual-tactile policies usually feed tactile observations directly into action prediction, but rarely model tactile deformation dynamics during action generation. In this paper, we introduce VT-WAM, a Visual-Tactile World Action Model that jointly learns future visual prediction, tactile deformation prediction, and action prediction within a unified flow matching framework. In particular, VT-WAM introduces (1) Asymmetric Mixture-of-Transformers (MoT) attention to bridge a first-frame visual anchor with temporal tactile dynamics, and (2) contact-gated Action-Visual-Tactile Attention Guidance (AVTAG) to encourage action queries to rely on tactile evidence during contact phases. Across six real-world contact-rich manipulation tasks, VT-WAM achieves a 71.67% average success rate, outperforming Fast-WAM by 26.67% and OmniVTLA by 35.84%. Ablations demonstrate that modeling tactile deformation dynamics and guiding contact-phase tactile attention are both important for contact-rich tasks. Project website: https://vt-wam.github.io/.
