Search papers, labs, and topics across Lattice.
This paper introduces Reflex, a novel three-stage framework for visually grounded reactive whole-body humanoid control, addressing the challenges of coordinating body movements, real-time object dynamics inference, and egocentric vision grounding. By decomposing the problem into learning whole-body control, then dynamics inference from delayed observations, and finally visual grounding of dynamics, Reflex enables humanoids to perform complex reactive tasks like catching thrown objects. The system achieves high catch success rates in simulation and demonstrates promising performance on a real humanoid robot, significantly outperforming prior methods that rely on less integrated approaches.
Humanoid robots can now catch thrown objects with 65% success from 2.5m using only egocentric RGB-D, a leap in reactive whole-body control.
Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers'success at the same distances.
UW