Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Diffusion Policy with Behavioral Advantage Correction (DPBAC) algorithm, which addresses the distribution shift issue in offline reinforcement learning by correcting the learned policy's Q-function using the behavior policy's Q-function. The authors theoretically analyze the convergence of their behavioral advantage corrected policy evaluation (BAC-PE) approach and derive an upper bound on the Q-function estimation error. Experimental results on D4RL tasks reveal that DPBAC outperforms existing state-of-the-art methods, demonstrating enhanced policy representation and reduced bias in Q-value estimation.
DPBAC not only corrects Q-value estimation biases but also leverages diffusion models for superior policy representation in offline reinforcement learning.
In offline reinforcement learning (RL), the distribution shift between behavioral data and the learned policy can lead to erroneous \emph{Q}-value estimation, thereby misguiding the direction of policy optimization. To address this issue, we develop a behavioral advantage corrected policy evaluation (BAC-PE) approach, which utilizes the \emph{Q}-function of the behavior policy to correct the learned policy's \emph{Q}-function, thus mitigating pessimistic conservatism and overestimation bias. Furthermore, the convergence of BAC-PE is analyzed theoretically, and an upper bound on the difference between the learned \emph{Q}-function and the true \emph{Q}-function is derived. To alleviate distribution shift, this work employs diffusion models to represent both the behavior policy and the learned policy, performing distribution matching for accurate policy regularization. Additionally, \emph{Q}-value guidance is incorporated into the training process to achieve effective policy improvement. By combining BAC-PE with diffusion policy modeling, we propose the diffusion policy with behavioral advantage correction (DPBAC) algorithm. Compared to existing offline methods, DPBAC demonstrates stronger policy representation capabilities and effectively mitigates the bias in \emph{Q}-value estimation. Experimental results on multiple domains of D4RL tasks show that DPBAC achieves superior performance, with notable advantages over state-of-the-art (SOTA) algorithms.