Search papers, labs, and topics across Lattice.
This paper introduces Robust Adversarial Policy Optimization (RAPO), a novel distributionally robust RL method that directly addresses dynamics uncertainty. RAPO employs a dual formulation to expose the robustness-performance trade-off, approximating the trajectory-level temperature parameter with an adversarial network and using Boltzmann reweighting over dynamics ensembles at the model level. Experiments demonstrate that RAPO outperforms existing robust RL baselines in resilience to uncertainty and generalization to out-of-distribution dynamics.
Steering worst-case trajectories with an adversarial network and Boltzmann reweighting dynamics ensembles yields a surprisingly stable and efficient approach to robust RL under dynamics uncertainty.
Reinforcement learning (RL) policies often fail under dynamics that differ from training, a gap not fully addressed by domain randomization or existing adversarial RL methods. Distributionally robust RL provides a formal remedy but still relies on surrogate adversaries to approximate intractable primal problems, leaving blind spots that potentially cause instability and over-conservatism. We propose a dual formulation that directly exposes the robustness-performance trade-off. At the trajectory level, a temperature parameter from the dual problem is approximated with an adversarial network, yielding efficient and stable worst-case rollouts within a divergence bound. At the model level, we employ Boltzmann reweighting over dynamics ensembles, focusing on more adverse environments to the current policy rather than uniform sampling. The two components act independently and complement each other: trajectory-level steering ensures robust rollouts, while model-level sampling provides policy-sensitive coverage of adverse dynamics. The resulting framework, robust adversarial policy optimization (RAPO) outperforms robust RL baselines, improving resilience to uncertainty and generalization to out-of-distribution dynamics while maintaining dual tractability.
BAIR