Search papers, labs, and topics across Lattice.
This paper introduces MPC-Injection, a method that enhances off-policy reinforcement learning for locomotion by incorporating transitions from a model predictive controller into the replay buffer, steering the learning process towards preferred gaits. The approach circumvents the complexities of reward shaping and adversarial imitation learning by directly influencing the state distribution without altering the task reward structure. Experiments demonstrate that MPC-Injection yields qualitatively superior gaits in both simulation and real-world applications, achieving results comparable to complex reward shaping methods while significantly reducing overhead.
MPC-Injection achieves usable locomotion gaits with minimal overhead, outperforming traditional reward shaping and imitation learning techniques.
Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred gait by inserting transitions into the replay buffer from a model predictive controller solving the same Markov decision process. Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. Instead, the controller's preferred behavior is transferred to the policy purely through the replay state distribution. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward, producing gaits qualitatively comparable to those of reward shaping with twenty-one tuned terms and of adversarial motion priors without their discriminator and retargeting overhead. We further analyze how the injected transitions bias actor-critic updates toward controller-visited states, allowing the policy to learn behaviors that pure RL may fail to reach under simple reward functions.
