Search papers, labs, and topics across Lattice.
MuxGel introduces a spatially multiplexed visuo-tactile sensor that interleaves tactile-sensitive and transparent regions using a checkerboard coating pattern, enabling simultaneous capture of external visual information and contact-induced tactile signals. A U-Net-based reconstruction framework, trained via sim-to-real transfer, is used to decouple and restore high-fidelity tactile and visual fields from the multiplexed input. Experiments on unseen objects and grasping tasks demonstrate MuxGel's generalization, accuracy, and enhanced perceptual capabilities compared to vision-based tactile sensors.
Finally, a GelSight-style sensor that doesn't force you to choose between pre-contact vision and high-fidelity tactile sensing.
High-fidelity visuo-tactile sensing is important for precise robotic manipulation. However, most vision-based tactile sensors face a fundamental trade-off: opaque coatings enable tactile sensing but block pre-contact vision. To address this, we propose MuxGel, a spatially multiplexed sensor that captures both external visual information and contact-induced tactile signals through a single camera. By using a checkerboard coating pattern, MuxGel interleaves tactile-sensitive regions with transparent windows for external vision. This design maintains standard form factors, allowing for plug-and-play integration into GelSight-style sensors by simply replacing the gel pad. To recover full-resolution vision and tactile signals from the multiplexed inputs, we develop a U-Net-based reconstruction framework. Leveraging a sim-to-real pipeline, our model effectively decouples and restores high-fidelity tactile and visual fields simultaneously. Experiments on unseen objects demonstrate the framework's generalization and accuracy. Furthermore, we demonstrate MuxGel's utility in grasping tasks, where dual-modality feedback facilitates both pre-contact alignment and post-contact interaction. Results show that MuxGel enhances the perceptual capabilities of existing vision-based tactile sensors while maintaining compatibility with their hardware stacks. Project webpage: https://zhixianhu.github.io/muxgel/.
