Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Solver-Gradient Guided Reinforcement Learning (SG-RL), which enhances the adaptation of cost-function weights in Model Predictive Control (MPC) by integrating solver-derived gradients into the reinforcement learning process. By maintaining the objective of maximizing closed-loop return while leveraging low-variance gradients from MPC, SG-RL achieves significant improvements in sample efficiency and stability. Experimental results on autonomous racing platforms demonstrate that SG-RL can achieve the best closed-loop return with up to 70.6% fewer samples compared to traditional Proximal Policy Optimization (PPO) and outperforms Gradient-Based Policy Learning (GB-PL) baselines by at least 54%.
Achieving up to 70.6% fewer samples for optimal closed-loop performance could revolutionize how we approach online adaptation in reinforcement learning.
In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.