Search papers, labs, and topics across Lattice.
The paper introduces HERO, a new paradigm for humanoid robot loco-manipulation that leverages large vision models for open-vocabulary understanding and a residual-aware end-effector (EE) tracking policy for precise control. The EE tracking policy combines inverse kinematics, a learned neural forward model, goal adjustment, and replanning to improve tracking accuracy. Experiments in both simulation and real-world environments demonstrate that HERO enables humanoid robots to reliably manipulate diverse objects on surfaces with varying heights, achieving a 3.2x reduction in EE tracking error.
Humanoid robots can now manipulate everyday objects in the wild with unprecedented reliability, thanks to a novel system that combines the generalization of large vision models with a highly accurate, simulation-trained end-effector tracking policy.
Visual loco-manipulation of arbitrary objects in the wild with humanoid robots requires accurate end-effector (EE) control and a generalizable understanding of the scene via visual inputs (e.g., RGB-D images). Existing approaches are based on real-world imitation learning and exhibit limited generalization due to the difficulty in collecting large-scale training datasets. This paper presents a new paradigm, HERO, for object loco-manipulation with humanoid robots that combines the strong generalization and open-vocabulary understanding of large vision models with strong control performance from simulated training. We achieve this by designing an accurate residual-aware EE tracking policy. This EE tracking policy combines classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, c) goal adjustment, and d) replanning. Together, these innovations help us cut down the end-effector tracking error by 3.2x. We use this accurate end-effector tracker to build a modular system for loco-manipulation, where we use open-vocabulary large vision models for strong visual generalization. Our system is able to operate in diverse real-world environments, from offices to coffee shops, where the robot is able to reliably manipulate various everyday objects (e.g., mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests in simulation and the real world demonstrate the effectiveness of our proposed design. We believe the advances in this paper can open up new ways of training humanoid robots to interact with daily objects.
