Search papers, labs, and topics across Lattice.
This paper introduces the Feature-Extracted Latent Tactile (FELT) framework, which generates tactile signals from RGB images to enhance visuo-tactile manipulation without the need for extensive tactile data collection. By employing a frozen visual encoder and a lightweight query decoder, FELT synthesizes per-finger pressure tactile images, allowing for effective manipulation in scenarios where tactile feedback is crucial but hard to obtain. Experiments across four manipulation tasks reveal that integrating generated tactile images and latent features significantly improves policy success compared to vision-only approaches, demonstrating the framework's practical utility in real-world applications.
Generating tactile signals from vision could revolutionize how robots manipulate objects in complex environments where touch is essential but hard to measure.
The sense of touch is central to manipulation, especially when vision is occluded or ambiguous. Although combining vision and touch improves manipulation, learning robust visuo-tactile policies requires substantial tactile data. Such data remains scarcer than visual data, because tactile sensors are fragile, specialized, and hard to standardize. To address this, we present Feature-Extracted Latent Tactile (FELT), a learning-based framework that synthesizes per-finger pressure tactile images from RGB observations, reducing the need for tactile-equipped data collection. FELT uses a large frozen visual encoder and a lightweight query decoder to predict tactile signals in a single feed-forward pass. To respect the physical topology of dual-finger tactile sensors, FELT decodes the left and right tactile sensor panels through separate branches, capturing the asymmetric contact patterns during interactions such as wiping, insertion, and in-hand rotation. At inference time, FELT only requires RGB data, allowing us to augment existing vision-only data with tactile observations, either as generated tactile images or as latent tactile features. Experiments on four contact-rich manipulation tasks demonstrate that both generated tactile images and latent tactile features improve policy success over vision-only baselines, with latent feature requiring no real tactile sensor during policy training or deployment. Supplementary material is available on our anonymous website: https://felt-tactile.github.io/.
