Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper introduces Loopless Variance-Reduced Policy Gradient (LVR-PG), a unified theoretical framework for first-order policy gradient estimators that combines recursive updates with defensive importance sampling. LVR-PG achieves $\tilde{\mathcal{O}}(\epsilon^{-2})$ last-iterate global convergence for general parametric policies under weak gradient domination, using only first-order updates and minimal assumptions. This result establishes minimax-optimality for policy gradient methods, matching state-of-the-art lower bounds.
First-order policy gradient methods can achieve minimax-optimal sample complexity for general parametric policies without restrictive assumptions or second-order operations.
Standard Policy Gradient (PG) algorithms suffer from high estimation variance because they rely on Monte Carlo updates. While off-policy variance reduction mitigates this problem, establishing optimal sample complexity currently requires either restrictive assumptions, such as uniformly bounded importance weights, or impractical second-order operations. In this work, we introduce Loopless Variance-Reduced PG (LVR-PG), a unified theoretical framework that abstracts a class of first-order estimators that integrate recursive updates with defensive importance sampling, including known estimators such as PAGE-PG and STORM-PG. We prove that this algorithmic class achieves $\tilde{\mathcal{O}}(\epsilon^{-2})$ last-iterate global convergence under weak gradient domination for general parametric policies. We establish this guarantee using purely first-order updates, with minimal assumptions and without gradient preconditioners. By extending state-of-the-art lower bounds to continuous parametric policies, we establish an $\Omega(\epsilon^{-2})$ sample complexity limit, confirming our framework is minimax-optimal up to logarithmic factors and horizon terms.