Search papers, labs, and topics across Lattice.
This paper enhances the theoretical understanding of dueling Q-learning by providing a direct interpretation of the centered tabular decomposition and establishing convergence guarantees for the unregularized, constant step-size recursion. The authors derive an exact switching linear system representation for deterministic dueling Q-learning and present a finite-time error bound for its stochastic counterpart. These findings clarify the distinct roles of value and advantage updates in improving learning efficiency within the Q-function framework.
Dueling Q-learning's efficiency gains are now backed by rigorous convergence guarantees, revealing how value and advantage updates function as distinct gains in the learning process.
Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown. The deep Q-network (DQN) extends Q-learning by using a deep neural network for Q-function approximation, which makes Q-learning applicable to more practical high-dimensional problems. Dueling Q-learning decomposes the Q-function into a value function and an advantage function and learns the two components jointly, which can improve learning efficiency. However, the theoretical understanding of dueling Q-learning is still limited. Recent work has initiated an analysis of tabular dueling Q-learning, but existing guarantees focus on a regularized formulation and leave the pure tabular update less completely understood. This paper strengthens that line of analysis by adding a direct interpretation of the centered tabular decomposition and by establishing convergence guarantees for the unregularized, unprojected constant step-size recursion. In particular, we derive an exact switching linear system representation for deterministic dueling Q-learning and a finite-time error bound in expectation for the sampled stochastic version. The analysis clarifies how the value and advantage updates act as different gains on the action-common (value function) and action-differential (advantage function) components of the Q-function.
