Search papers, labs, and topics across Lattice.
This paper addresses the high variance in online policy evaluation for reinforcement learning by introducing a double-loop gradient-based algorithm that learns behavior policies robust to transition uncertainties. The proposed method derives novel transition-variance gradient expressions and guarantees global convergence, thus enhancing the reliability of policy evaluation in real-world applications. Empirical results show that this approach significantly reduces sensitivity to transition perturbations compared to existing methods, highlighting its practical advantages.
Behavior policies trained in simulation can be robust to real-world transition uncertainties, reducing evaluation variance and reliance on costly real-world samples.
In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not account for uncertainties in the transition functions. In practice, simulator transitions often differ from the real world due to modeling errors or approximation limitations. As a result, behavior policies trained in simulation may still yield high variance when deployed in real environments, leading to costly reliance on real-world evaluation samples. In this work, we propose a double-loop gradient-based algorithm for learning behavior policies that are both efficient and robust to transition uncertainty. Theoretically, we derive novel transition-variance gradient expressions and establish global convergence guarantees for the algorithm. Numerically, we demonstrate that our method is less sensitive to transition perturbations than existing approaches, providing supportive evidence for its practical utility.
