Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ViTaS, a visuomotor learning framework that leverages both visual and tactile information through Soft Fusion Contrastive Learning and a CVAE module to improve performance in manipulation tasks, especially in occluded scenarios. The Soft Fusion Contrastive Learning method is designed to better exploit the alignment and complementarity of visual and tactile representations. Experiments across 12 simulated and 3 real-world environments demonstrate that ViTaS significantly outperforms existing baselines, highlighting the benefits of the proposed fusion and contrastive learning approach.
Overcome visual occlusion in robotic manipulation by fusing vision and touch with contrastive learning, outperforming standard concatenation methods.
Tactile information plays a crucial role in human manipulation tasks and has recently garnered increasing attention in robotic manipulation. However, existing approaches mostly focus on the alignment of visual and tactile features and the integration mechanism tends to be direct concatenation. Consequently, they struggle to effectively cope with occluded scenarios due to neglecting the inherent complementary nature of both modalities and the alignment may not be exploited enough, limiting the potential of their real-world deployment. In this paper, we present ViTaS, a simple yet effective framework that incorporates both visual and tactile information to guide the behavior of an agent. We introduce Soft Fusion Contrastive Learning, an advanced version of conventional contrastive learning method and a CVAE module to utilize the alignment and complementarity within visuo-tactile representations. We demonstrate the effectiveness of our method in 12 simulated and 3 real-world environments, and our experiments show that ViTaS significantly outperforms existing baselines. Project page: https://skyrainwind.github.io/ViTaS/index.html.