Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a stochastic differential equation (SDE) approximation for linear temporal difference (TD) learning under Markovian noise, addressing the limitations of the classical ordinary differential equation (ODE) approach that overlooks stochastic fluctuations. By distinguishing the contraction dynamics of the projected Bellman operator from the effects of Markovian sampling, the authors elucidate the relationship between long-run covariance and the error floor associated with constant-step size learning. The key finding reveals that the interplay between these factors is critical for understanding the performance limits of TD learning in noisy environments.
The new SDE model reveals how Markovian noise fundamentally shapes the error floor in TD learning, challenging traditional views based solely on deterministic dynamics.
Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.
TU Munich