Search papers, labs, and topics across Lattice.
This paper investigates the variance characteristics of temporal difference (TD) learning in a phased setting with tabular representation, revealing that TD's variance reduction stems from its ability to aggregate multiple independent trajectories. The authors establish that TD's variance is asymptotically bounded by that of Monte Carlo (MC) estimators and demonstrate that shorter horizon updates lead to reduced variance for a fixed sample size. Additionally, they present Direct Advantage Estimation (DAE) as a regression-adjusted control variate that offers a tighter variance bound than TD in large-sample scenarios, supported by numerical experiments in controlled environments.
TD learning can achieve variance bounds comparable to Monte Carlo methods, but with the added advantage of shorter updates leading to more stable estimates.
We analyze the variance of temporal difference (TD) learning using the phased setting with tabular representation, and show that one of the mechanisms behind its ability to reduce variance is by effectively aggregating over a larger number of independent trajectories. Based on this insight, we demonstrate that (1) the variance of TD is asymptotically bounded from above by Monte Carlo (MC) estimators, and (2) shorter horizon updates incurs less variance for a fixed number of samples. Beyond TD, we show that Direct Advantage Estimation (DAE), a method for estimating the advantage function, can be seen as a type of regression-adjusted control variate, which achieves a tighter bound on the variance compared to TD in the large-sample limit. Finally, we numerically illustrate the behaviors of these estimators with carefully designed environments.
ETH