Search papers, labs, and topics across Lattice.
CReF is introduced as a single-stage depth-conditioned humanoid locomotion framework that learns directly from raw depth data, avoiding explicit geometric abstractions. It uses proprioception-queried cross-modal attention to fuse proprioception and depth tokens, integrates this representation temporally with a GRU, and incorporates a terrain-aware foothold placement reward. Experiments demonstrate robust traversal over diverse terrains in simulation and effective zero-shot transfer to real-world scenes.
Humanoids can now traverse complex real-world terrain, including handrails and cluttered environments, with zero-shot transfer from simulation, thanks to a novel depth-conditioned locomotion framework.
Stable traversal over geometrically complex terrain increasingly requires exteroceptive perception, yet prior perceptive humanoid locomotion methods often remain tied to explicit geometric abstractions, either by mediating control through robot-centric 2.5D terrain representations or by shaping depth learning with auxiliary geometry-related targets. Such designs inherit the representational bias of the intermediate or supervisory target and can be restrictive for vertical structures, perforated obstacles, and complex real-world clutter. We propose CReF (Cross-modal and Recurrent Fusion), a single-stage depth-conditioned humanoid locomotion framework that learns locomotion-relevant features directly from raw forward-facing depth without explicit geometric intermediates. CReF couples proprioception and depth tokens through proprioception-queried cross-modal attention, fuses the resulting representation with a gated residual fusion block, and performs temporal integration with a Gated Recurrent Unit (GRU) regulated by a highway-style output gate for state-dependent blending of recurrent and feedforward features. To further improve terrain interaction, we introduce a terrain-aware foothold placement reward that extracts supportable foothold candidates from foot-end point-cloud samples and rewards touchdown locations that lie close to the nearest supportable candidate. Experiments in simulation and on a physical humanoid demonstrate robust traversal over diverse terrains and effective zero-shot transfer to real-world scenes containing handrails, hollow pallet assemblies, severe reflective interference, and visually cluttered outdoor surroundings.
