Search papers, labs, and topics across Lattice.
The Heterogeneous Tactile Transformer (HTT) addresses the challenge of transferring tactile learning across diverse sensors by developing a framework that combines sensor-specific encoders with a shared transformer trunk. By utilizing a novel Heterogeneous Paired Tactile (HPT) dataset with 1.6M synchronized frames, HTT is pretrained to learn shared tactile representations through masked reconstruction and cross-modal alignment. The results demonstrate that HTT can effectively adapt to new tasks and previously unseen sensors, significantly enhancing the scalability of contact-rich manipulation policies.
HTT enables tactile learning across diverse sensors, achieving adaptability that was previously unattainable in contact-rich manipulation tasks.
Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors. Dataset, code, and model checkpoints will be released upon publication at https://jxbi1010.github.io/htt-gh-page/.
CMU ML
NUS