Search papers, labs, and topics across Lattice.
VIP-Loco combines vision-based scene understanding with reinforcement learning and model predictive control for legged robot locomotion. An internal model maps proprioceptive states and depth images into kinodynamic features used by an RL policy during training. This learned model is then deployed within an infinite-horizon MPC formulation, enabling adaptation and structured planning. VIP-Loco demonstrates robust locomotion across diverse terrains and robot morphologies in simulation, outperforming state-of-the-art methods.
Legged robots can now nimbly navigate complex terrains by fusing learned visual perception with infinite-horizon planning, achieving robustness beyond traditional MPC or RL approaches.
Perceptive locomotion for legged robots requires anticipating and adapting to complex, dynamic environments. Model Predictive Control (MPC) serves as a strong baseline, providing interpretable motion planning with constraint enforcement, but struggles with high-dimensional perceptual inputs and rapidly changing terrain. In contrast, model-free Reinforcement Learning (RL) adapts well across visually challenging scenarios but lacks planning. To bridge this gap, we propose VIP-Loco, a framework that integrates vision-based scene understanding with RL and planning. During training, an internal model maps proprioceptive states and depth images into compact kinodynamic features used by the RL policy. At deployment, the learned models are used within an infinite-horizon MPC formulation, combining adaptability with structured planning. We validate VIP-Loco in simulation on challenging locomotion tasks, including slopes, stairs, crawling, tilting, gap jumping, and climbing, across three robot morphologies: a quadruped (Unitree Go1), a biped (Cassie), and a wheeled-biped (TronA1-W). Through ablations and comparisons with state-of-the-art methods, we show that VIP-Loco unifies planning and perception, enabling robust, interpretable locomotion in diverse environments.
