Search papers, labs, and topics across Lattice.
This study introduces NoContactNoWorries, a transformer-based multimodal framework that enables robots to estimate contact states during in-hand manipulation by integrating RGB-D visual data with proprioceptive feedback. By mimicking human perceptual abilities, the framework provides a scalable alternative to traditional tactile sensors, addressing issues of cost and fragility. Validation through training on diverse objects demonstrates that the inferred contact signal enhances the performance of reinforcement learning agents in object reorientation tasks, successfully generalizing to new objects in both simulated and real-world environments.
Robots can now infer contact states from vision and proprioception, eliminating the need for costly tactile sensors.
Perceiving physical contact is fundamental to dexterous manipulation. While robots often rely on dedicated hardware tactile sensors, humans exhibit a remarkable ability to infer contact by integrating visual information with an innate sense of their body's pose and movement. Inspired by this embodied perceptual skill, we investigate whether a robot can learn to infer contact from vision, an approach that also offers a scalable alternative to tactile hardware specifically for binary contact estimation, which faces practical challenges in cost, fragility, and integration. We present NoContactNoWorries, a transformer-based multimodal framework that fuses RGB-D vision with the robot's proprioception to infer binary contact states as a pseudo-tactile signal for hand-object interactions. We validate by training a single contact prediction model on multiple objects and show that the inferred contact signal supports downstream reinforcement learning agents for in-hand object reorientation, generalizing to novel objects. Experiments in both simulation and on a real-world robot validate our approach, highlighting the feasibility of inferring contact from vision and proprioception. Project Page: https://soham2560.github.io/no-contact-no-worries/
