Search papers, labs, and topics across Lattice.
This paper addresses the off-policy evaluation (OPE) challenge by introducing a distributional approach that estimates the entire return distribution rather than just its expectation. The authors present a novel algorithm, Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE), which leverages deep quantile process regression to achieve this goal. Their findings indicate that DQPOPE not only maintains the sample efficiency of traditional methods but also offers significantly improved precision and robustness in policy value estimates, thereby enhancing the effectiveness of distributional reinforcement learning techniques.
DQPOPE achieves the same sample efficiency as traditional OPE methods while providing a comprehensive return distribution, leading to significantly more accurate policy evaluations.
This paper investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of the total return, as in most existing OPE methods, we aim to estimate the entire return distribution. To this end, we introduce a quantile-based approach for OPE using deep quantile process regression, presenting a novel algorithm called Deep Quantile Process regression-based Off-Policy Evaluation (DQPOPE). We provide new theoretical insights into the deep quantile process regression technique, extending existing approaches that estimate discrete quantiles to estimate a continuous quantile function. A key contribution of our work is the rigorous sample complexity analysis for distributional OPE with deep neural networks, bridging theoretical analysis with practical algorithmic implementations. We show that DQPOPE achieves statistical advantages by estimating the full return distribution using the same sample size required to estimate a single policy value using conventional methods. Empirical studies further show that DQPOPE provides significantly more precise and robust policy value estimates than standard methods, thereby enhancing the practical applicability and effectiveness of distributional reinforcement learning approaches.
