Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces RGB-S, a framework that enhances robotic dexterous manipulation by explicitly aligning tactile sensor data with visual representations using physical contact priors. By projecting tactile sensor locations onto RGB images and generating force-modulated Gaussian saliency maps, the method effectively addresses the challenges of data inefficiency and poor generalization in existing approaches. The results demonstrate a significant improvement in manipulation success rates under severe visual occlusions, with a 26.7 percentage point increase over the best implicit baseline, highlighting the framework's robustness and spatial reasoning capabilities.
Explicitly grounding tactile data in the image domain boosts manipulation success rates by over 26% in occluded environments, challenging the status quo of implicit learning methods.
Effective visuo-tactile integration is critical for robotic dexterous manipulation, especially when visual observations are unreliable or occluded. However, robustly aligning sparse, heterogeneous tactile measurements with dense visual representations remains a fundamental challenge. Most existing approaches require policies to learn cross-modal correspondences implicitly from limited demonstrations, without leveraging geometric priors. As a result, they are often data-inefficient and generalize poorly when visual observations are degraded. To address this limitation, we propose a framework that explicitly grounds physical contacts in the image domain. Using robot forward kinematics and camera calibration, we project tactile sensor locations directly onto the RGB image plane. We then render force-modulated Gaussian saliency maps to model spatial uncertainty arising from kinematic and calibration errors. By integrating these 2D spatial anchors through a zero-initialized conditioning architecture, our method injects physical contact priors into standard visual backbones while preserving pre-trained visual representations. We evaluate our method on six dexterous manipulation tasks in both simulation and the real world under severe visual occlusions. Real-world experiments show that explicit RGB-S grounding in the image domain improves real-world occluded manipulation success rates by $26.7$ percentage points over the strongest implicit visuo-tactile baseline, suggesting its improved spatial reasoning and robustness to occlusion. Project page: touch-as-saliency.github.io