Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
We analyze the convergence of semi-gradient TD learning with two-layer neural networks in the mean-field regime, identifying that entropy-induced relaxation can dominate bootstrap target sensitivity to ensure stability. Under this condition, we prove exponential convergence to a unique fixed point and quantify the resulting bias. This stability mechanism is shown to extend to entropy-regularized off-policy control via soft Q-learning.
Semi-gradient TD learning with neural networks is provably stable and exponentially converges under a simple entropy-induced relaxation condition.
Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability. We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target. Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution. We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.
RIKEN