Search papers, labs, and topics across Lattice.
This paper systematically analyzes the instability in Deep Q-learning (DQL) by examining operator-level biases, estimator-level sensitivities, and parameter-dynamics imbalances. The authors identify a self-reinforcing trap triggered by rewards and propose stabilization techniques, including controlled bootstrapping and ensemble quantile estimation, to mitigate these issues. Experimental results on Atari-100K and Procgen show that these methods lead to improved training stability and competitive performance compared to existing approaches.
A self-reinforcing instability trap in Deep Q-learning can be effectively mitigated through controlled bootstrapping and ensemble quantile estimation, leading to more stable training outcomes.
Deep Q-learning (DQL) has achieved remarkable empirical success in reinforcement learning, yet its training process remains notoriously unstable. Existing studies often attribute instability to isolated factors such as overestimation bias or representation learning issues, lacking a unified understanding of how different sources of instability interact during recursive value estimation. In this work, we provide a systematic analysis of instability in deep Q-learning from three complementary perspectives: operator-level bias in Bellman bootstrapping, estimator-level sensitivity of greedy action selection to regression noise, and parameter-dynamics imbalance under aggressive data reuse. We identify a reward-triggered self-reinforcing trap and characteristic parameter spike dynamics, then derive stabilization principles for controlled bootstrapping, ensemble quantile estimation, and spike-based parameter regulation. Experiments on Atari-100K and Procgen demonstrate competitive performance and improved training stability.
