Search papers, labs, and topics across Lattice.
This paper introduces Humanoid Transformer with Touch Dreaming (HTD), a multimodal Transformer trained with behavioral cloning and augmented with a novel "touch dreaming" objective that predicts future hand-joint forces and tactile latents. HTD is trained on data collected via a VR-based teleoperation system with human-to-humanoid motion mapping. Results across five contact-rich manipulation tasks demonstrate a 90.9% relative improvement in average success rate compared to a strong baseline, with latent-space tactile prediction providing a 30% relative gain.
Humanoid robots can now perform complex, contact-rich manipulation tasks with significantly improved dexterity and success by "dreaming" about the feel of their actions.
Humanoid robots promise general-purpose assistance, yet real-world humanoid loco-manipulation remains challenging because it requires whole-body stability, dexterous hands, and contact-aware perception under frequent contact changes. In this work, we study dexterous, contact-rich humanoid loco-manipulation. We first develop an RL-based whole-body controller that provides stable lower-body and torso execution during complex manipulation. Built on this controller, we develop a whole-body humanoid data collection system that combines VR-based teleoperation with human-to-humanoid motion mapping, enabling efficient collection of real-world demonstrations. We then propose Humanoid Transformer with Touch Dreaming (HTD), a multimodal encoder--decoder Transformer that models touch as a core modality alongside multi-view vision and proprioception. HTD is trained in a single stage with behavioral cloning augmented by touch dreaming: in addition to predicting action chunks, the policy predicts future hand-joint forces and future tactile latents, encouraging the shared Transformer trunk to learn contact-aware representations for dexterous interaction. Across five contact-rich tasks, Insert-T, Book Organization, Towel Folding, Cat Litter Scooping, and Tea Serving, HTD achieves a 90.9% relative improvement in average success rate over the stronger baseline. Ablation results further show that latent-space tactile prediction is more effective than raw tactile prediction, yielding a 30% relative gain in success rate. These results demonstrate that combining robust whole-body execution, scalable humanoid data collection, and predictive touch-centered learning enables versatile, high-dexterity humanoid manipulation in the real world. Project webpage: humanoid-touch-dream.github.io.
CMU ML