Search papers, labs, and topics across Lattice.
This paper introduces a corrected heavy-ball Q-learning method that enhances convergence rates in reinforcement learning by leveraging a switched linear system representation. The authors establish theoretical guarantees for faster convergence compared to standard Q-learning and extend these findings to scenarios involving linear function approximation. By employing the joint spectral radius of the associated switching families, the study provides novel insights into the role of heavy-ball momentum in accelerating Q-learning processes.
Heavy-ball Q-learning can converge faster than traditional methods, offering a new lens on acceleration in reinforcement learning.
This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning. The same construction is then extended to Q-learning with linear function approximation, where analogous convergence and acceleration statements are derived. The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and on the joint spectral radius (JSR) of the associated switching families. This SLS viewpoint is not commonly used in standard analyses of Q-learning, and it provides a complementary framework and new insight into how heavy-ball momentum can accelerate Q-learning.
