Search papers, labs, and topics across Lattice.
This paper introduces Warp RL, a novel policy adaptation method that enhances residual reinforcement learning by utilizing an invertible, state-conditioned transformation of the base policy's action distribution. The authors identify critical limitations of traditional additive corrections, such as miscalibrated confidence and wrong variance, which can lead to underperformance in dynamically shifted environments. Through extensive experiments on ManiSkill3 manipulation tasks and a real-robot peg-insertion task, Warp RL not only matches but significantly outperforms traditional methods when distributional reshaping is necessary, demonstrating its effectiveness in adapting to complex dynamics shifts.
Traditional residual corrections can falter under dynamics shifts, but Warp RL reshapes action distributions to achieve superior performance in challenging environments.
Residual reinforcement learning adapts a pretrained robot policy by learning an additive correction to its actions. While effective when adaptation amounts to shifting the base policy's action distribution, additive corrections cannot change the distribution's shape, scale, or state-dependent geometry -- limitations we formalize as wrong variance, miscalibrated confidence, and non-uniform correction. We show that these matter under dynamics shift: when the base distribution is geometrically mismatched to the shifted system, residual correction can underperform even the unadapted policy. We propose \textbf{Warp RL}, a policy adaptation method that replaces additive residuals with an invertible, state-conditioned transformation of the base policy's action distribution. Instantiated with monotonic rational-quadratic spline flows [arXiv:0706.1234v1], Warp RL preserves identity initialization, strictly generalizes additive residual correction, and exposes a structured adaptation space suitable for both policy-gradient and gradient-free optimization. Across a variety of ManiSkill3 manipulation tasks with controlled dynamics shifts, Warp RL matches residual correction when translation is sufficient and substantially outperforms it when adaptation requires distributional reshaping. We further demonstrate that warping can replace additive correction in an off-policy sim-to-real pipeline, achieving comparable success rate with 30% faster task completion on a real-robot peg-insertion task.
