Search papers, labs, and topics across Lattice.
This paper establishes mean-square and concentration bounds for stochastic approximation (SA) with arbitrary norm contractive mappings, particularly under a multiplicative noise model relevant to reinforcement learning. The authors provide a unified and elementary analysis that circumvents the need for smoothing techniques by directly applying a one-step Lyapunov drift inequality and a probabilistic induction method. Key results include the first sub-Gaussian tailed maximal concentration bound for SA with multiplicative noise, demonstrating that the stepsize can depend logarithmically on the confidence level, thus enhancing the robustness of SA algorithms in practice.
Achieving the first sub-Gaussian concentration bound for stochastic approximation under multiplicative noise could revolutionize how we approach stability in reinforcement learning algorithms.
We establish mean-square and concentration bounds for stochastic approximation (SA) with arbitrary norm contractive mappings, under a multiplicative noise model where the noise may scale affinely with the norm of the iterates, and the iterates are potentially unbounded. These settings arise in reinforcement learning, where operators are often contractive in the $\ell_\infty$ norm and the noise scales with the iterates. To address the arbitrary norm, earlier works replace the non-smooth squared norm with a smooth Lyapunov function constructed via the generalized Moreau envelope. For concentration analysis, these works handle multiplicative noise and unbounded iterates through a multi-stage bootstrapping argument that starts from a time-varying worst-case bound and iteratively refines it. We instead present a unified and elementary analysis that yields both bounds. Using an averaged noise sequence and corresponding auxiliary iterates, we obtain a one-step Lyapunov drift inequality for the normed error directly, without smoothing the norm or constructing an envelope. For the mean-square bound, we combine this drift inequality with an induction argument showing that the iterates remain bounded in expectation. For the concentration bound, we develop a probabilistic induction over a sequence of "good" events on which the iterates are controlled, allowing the standard Azuma-Hoeffding bound to be applied. Our approach yields the first sub-Gaussian tailed maximal (all-time) concentration bound for SA under multiplicative noise, by allowing the stepsize to depend logarithmically on the confidence level. Beyond the specific setting considered here, we discuss the generalizability of these proof techniques to other noise models and iterative algorithms.
