Search papers, labs, and topics across Lattice.
This paper introduces CAAT, a contact-aware framework that enhances visuo-tactile manipulation by integrating explicit contact priors through attention scaling and dynamic tactile masking. By emphasizing visual information pre-contact and tactile information during contact, CAAT significantly improves cross-modal representation learning from demonstrations. In both simulation and real-world experiments, CAAT outperforms traditional visuo-tactile fusion methods, achieving up to 21.1 percentage points higher success rates across various policy architectures.
Explicitly incorporating contact priors into visuo-tactile policies can boost manipulation success rates by over 21% in real-world scenarios.
In contact-rich manipulation, visual observations primarily guide motion in free space, whereas tactile observations become particularly informative during contact. However, standard Transformer-based visuo-tactile policies typically rely on either token concatenation or learnable gating. These approaches lack explicit contact-aware priors, making it difficult to efficiently learn effective cross-modal representations from demonstrations. To address this limitation, we propose CAAT, a lightweight contact-aware framework that explicitly incorporates contact priors through attention scaling and dynamic tactile masking. Specifically, CAAT emphasizes visual information before contact and tactile information during contact. It also suppresses static background tokens by comparing the current tactile observation with a non-contact reference. CAAT can be integrated into commonly used Transformer-based policies without modifying their action decoders. In simulation, integrating CAAT with ACT improves the average success rate by 18.0 percentage points over direct visuo-tactile fusion and by 10.0 percentage points over gated fusion. In real-world experiments using a visuo-tactile UMI platform, CAAT achieves an average success rate of 60.0% across ACT, Diffusion Policy, and $π_0$, outperforming the strongest baseline by an average of 21.1 percentage points. These results demonstrate that explicit contact priors and dynamic tactile masking are effective in improving visuo-tactile policy learning and task performance of diverse policy architectures. https://mrjiangjm.github.io/caat/
