Search papers, labs, and topics across Lattice.
This survey synthesizes the mathematical foundations of reinforcement learning (RL) by organizing key concepts from probability, optimization, and operator theory. It emphasizes the significance of Markov decision processes (MDPs), contraction mappings, and fixed-point theory in establishing convergence guarantees for various RL algorithms. Additionally, it explores advanced topics such as off-policy evaluation, constrained RL, and the role of function approximation, providing a comprehensive framework for researchers in related fields to engage with RL methodologies.
Unifying diverse mathematical frameworks reveals critical insights into convergence and performance guarantees for reinforcement learning algorithms.
Reinforcement learning (RL) is increasingly grounded in tools from probability, optimization, and operator theory. This survey organizes the mathematical structures that underpin the design and analysis of modern algorithms in RL. We begin from Markov decision processes (MDPs) and the Bellman operators, emphasizing contraction mappings, monotonicity, and fixed-point theory that yield convergence guarantees and rates for value and policy iteration, and temporal-difference schemes. We then develop the optimization perspective: stochastic approximation and martingale methods, convex duality and the role of regularization linking mirror/proximal methods. Function approximation is treated through linear and non-linear settings, covering stabilization, error decomposition, and sample-complexity via concentration inequalities for dependent data and mixing processes. We further cover off-policy evaluation/learning, constrained RL and constrained MDPs (CMDPs). Throughout we unify algorithmic templates under common operator and variational lenses, highlighting both finite-sample bounds and asymptotic results. Our presentation is intended to provide a unified mathematical entry point for researchers in probability, optimization, and statistics interested in reinforcement learning.
