Search papers, labs, and topics across Lattice.
This paper formulates offline reinforcement learning as a robust policy optimization problem, explicitly addressing transition uncertainty by treating the transition kernel as a decision variable within an uncertainty set. They propose Robust Regularized Policy Iteration (RRPI), which uses a KL-regularized surrogate to make the intractable max-min objective tractable, leading to an efficient policy iteration procedure. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance and exhibits robust behavior by learning Q-values that decrease in regions with higher epistemic uncertainty.
Offline RL can be made more robust to distribution shift by directly optimizing against worst-case transition dynamics within an uncertainty set, leading to policies that avoid unreliable out-of-distribution actions.
Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a $\gamma$-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods such as PMDB on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust behavior. The learned $Q$-values decrease in regions with higher epistemic uncertainty, suggesting that the resulting policy avoids unreliable out-of-distribution actions under transition uncertainty.
