Search papers, labs, and topics across Lattice.
This paper introduces GenHOI, a novel framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by imitating a single generated video without the need for task-specific training. By reconstructing the robot-object scene and analyzing generated videos to extract contact events and hand-object contact regions, GenHOI converts visual cues into geometric constraints that guide the optimization of interaction trajectories. The framework was validated through extensive simulations and real-world experiments, demonstrating its effectiveness across various complex tasks such as box grasping and bimanual chair carrying.
GenHOI allows humanoid robots to learn complex object interactions from a single video, eliminating the need for extensive training or demonstrations.
Humanoid-Object Interaction (HOI) is a fundamental capability for humanoid robots, yet it remains challenging due to the tight coupling between dynamic balance and stable interaction with diverse objects. Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay, which limits their ability to accommodate novel interaction scenarios. In this work, we present \textit{GenHOI}, a simple yet effective framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner by directly imitating a single generated video, without task-specific training or physical demonstration data. GenHOI first reconstructs the robot-object scene in simulation and renders a first-frame image, which, together with the language command, conditions the synthesis of a task-oriented interaction video. The generated video is then analyzed to identify interaction-relevant contact events and estimate hand-object contact regions, which are encoded as object-centric geometric constraints that convert visual interaction cues into physically grounded optimization priors. Guided by these priors, the reference motion recovered from the video is refined and smoothed to resolve the scale ambiguity inherent in 2D video generation, while adapting a single reference trajectory to unseen robot-object relative poses. The optimized trajectory is finally executed by a closed-loop tracking controller. We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks, including box grasping, asymmetric bimanual chair carrying, table lifting from below, and cylindrical-object enveloping.
