Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Collaborative Weighting Actor-Critic (CWAC) framework, which addresses the persistent overestimation bias in off-policy reinforcement learning by incorporating predictive uncertainty into value estimation. By employing a distributional critic and a collaborative weighting mechanism, CWAC effectively reweights TD-errors and uncertainty, allowing for more reliable learning from samples while mitigating the impact of noisy updates. Empirical evaluations show that CWAC significantly improves performance in various continuous control tasks when integrated with existing algorithms like SAC, TD3, and DDPG.
Overcoming the pervasive overestimation bias in off-policy reinforcement learning could lead to more stable and efficient training processes across diverse applications.
Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targets, resulting in non-stationary optimization, while greedy policy updates amplify early-stage estimation errors. The recursive propagation of such errors leads to persistent overestimation bias and degraded training stability in actor-critic methods. Existing approaches attempt to alleviate this issue via prioritized sampling or modified value learning objectives, but often overemphasize high-uncertainty transitions caused by limited data coverage or bootstrapping errors, thereby further amplifying bias.In this paper, we propose Collaborative Weighting Actor-Critic (CWAC), a unified framework that explicitly accounts for predictive uncertainty in value estimation. CWAC employs distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD-errors and uncertainty, enabling robust learning from reliable samples while suppressing noisy updates. In addition, we incorporate a stochastic pessimistic value estimation scheme via sampling from the return distribution, which effectively mitigates error propagation during policy improvement. CWAC can be seamlessly integrated into existing off-policy algorithm frameworks such as SAC, TD3, and DDPG with minimal overhead. Empirical results demonstrate that our proposed method significantly enhances performance across a diverse range of simulated tasks. Our code is publicly available at https://anonymous.4open.science/r/CWAC-348E.