Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the $λ$-target update method for linear Q-learning, which stabilizes learning by averaging periodic target updates with geometric weights. By analyzing this approach through a switching-system model, the authors demonstrate that varying $λ$ allows for a smooth transition between one-period updates and projected Q-value iteration. The results indicate that this method enhances stability in Q-learning, particularly when using linear function approximation, offering a more robust framework for reinforcement learning applications.
Averaging periodic target updates with geometric weights can significantly enhance stability in linear Q-learning, bridging the gap between simple and complex update strategies.
Periodic hard target updates are among the most common stabilization devices in modern deep Q-learning. Recent studies suggest that target updates can improve stability in Q-learning with function approximation, including linear function approximation. We introduce and analyze the so-called $λ$-target update, obtained by averaging the $m$-periodic target update maps with $λ$-geometric weights $(1-λ)λ^{m-1}$, $λ\in [0,1]$. The endpoint $λ=0$ recovers the one-period target update, while the continuous endpoint $λ\uparrow1$ recovers projected Q-value iteration. We study this mechanism for Q-learning with linear function approximation, namely linear Q-learning, using a switching-system model and related tools. For clarity, the paper treats a deterministic version; the formulation extends to stochastic reinforcement-learning settings.