Search papers, labs, and topics across Lattice.
The paper introduces HEX, a framework for whole-body coordinated manipulation on humanoid robots using a humanoid-aligned state representation and a Mixture-of-Experts Unified Proprioceptive Predictor trained on multi-embodiment data. HEX incorporates lightweight history tokens to efficiently capture temporal visual context and uses a residual-gated fusion mechanism to integrate visual-language cues with proprioceptive dynamics. Real-world experiments demonstrate state-of-the-art performance in task success rate and generalization, especially in fast-reaction and long-horizon scenarios.
Achieve human-like dexterity in humanoid robots by unifying visual-language cues with learned whole-body proprioceptive dynamics, outperforming prior methods in complex manipulation tasks.
Humans achieve complex manipulation through coordinated whole-body control, whereas most Vision-Language-Action (VLA) models treat robot body parts largely independently, making high-DoF humanoid control challenging and often unstable. We present HEX, a state-centric framework for coordinated manipulation on full-sized bipedal humanoid robots. HEX introduces a humanoid-aligned universal state representation for scalable learning across heterogeneous embodiments, and incorporates a Mixture-of-Experts Unified Proprioceptive Predictor to model whole-body coordination and temporal motion dynamics from large-scale multi-embodiment trajectory data. To efficiently capture temporal visual context, HEX uses lightweight history tokens to summarize past observations, avoiding repeated encoding of historical images during inference. It further employs a residual-gated fusion mechanism with a flow-matching action head to adaptively integrate visual-language cues with proprioceptive dynamics for action generation. Experiments on real-world humanoid manipulation tasks show that HEX achieves state-of-the-art performance in task success rate and generalization, particularly in fast-reaction and long-horizon scenarios.
