Search papers, labs, and topics across Lattice.
HALOMI is a novel framework that leverages human demonstrations to enhance humanoid loco-manipulation through active perception, addressing the challenges of world-frame tracking and human-to-humanoid gaps. By integrating egocentric sensing and a manifold-constrained controller, HALOMI enables robust head-hand tracking and aligns ego-view observations with humanoid actions. The framework was validated on a Unitree G1 humanoid robot, achieving an impressive average success rate of 85% across various tasks, including navigation and dynamic manipulation.
HALOMI bridges the human-to-humanoid gap, achieving 85% success in complex loco-manipulation tasks by leveraging active perception from human demonstrations.
Human demonstrations, which can be collected at scale and naturally capture active hand-eye coordination, are a promising data source for learning humanoid loco-manipulation. However, directly transferring human demonstrations to humanoids requires a precise world-frame tracking controller, which is often brittle under Out-of-Distribution(OOD) targets, while human-to-humanoid gaps persist in both egocentric observation and action execution. To address these challenges, we present HALOMI, a scalable framework for learning humanoid loco-manipulation with active perception from human demonstrations. HALOMI extends Universal Manipulation Interface (UMI) with egocentric sensing to collect ego-view and wrist-view observations along with head-hand trajectories at scale. We further propose a manifold-constrained controller that plans in a learned latent behavior manifold to enable precise and robust head-hand tracking in the world frame. To bridge the human-to-humanoid gap, we perform ego-view alignment and introduce a controller-aware reference trajectory adaptation to reduce mismatch in both observation and action execution. We validate HALOMI on a Unitree G1 humanoid robot with an actuated neck across five real-world tasks involving navigation, grasping, bimanual manipulation, whole-body coordination, and dynamic behaviors. Across the three quantitatively evaluated tasks, HALOMI achieves an average success rate of 85\%, while additional qualitative demonstrations show its ability to support dynamic tossing and deep-squat grasping.
