Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper provides a theoretical analysis of Soft Actor-Critic (SAC) when actor updates track an evolving policy mirror descent (PMD) target rather than the standard Gibbs distribution. Closing a major gap between SAC theory and practice where actors execute only few gradient steps per critic update, the work derives conditions for strong convexity and smoothness of the actor objective via the curvature of the continuous Q-function under the Legendre differential operator. The authors establish an $\mathcal{O}(N^{-1/5})$ finite-time best-iterate convergence rate up to approximation errors, proving that the PMD step size directly bounds target drift and eliminates the persistent, non-vanishing tracking error found in classical Gibbs formulations.
Standard Soft Actor-Critic suffers from an irreducible policy tracking error under conventional Gibbs targets, but switching to a policy mirror descent formulation enables direct control of target drift to guarantee provable $\mathcal{O}(N^{-1/5})$ convergence.
Soft Actor-Critic (SAC) is widely used for entropy-regularised reinforcement learning with continuous action spaces, and practical implementations perform only a few actor steps towards an evolving target. In this work, we prove convergence guarantees when the target policy arises from policy mirror descent and compare it with the classical Gibbs target. We derive sufficient conditions for the strong convexity and smoothness of the actor objective, characterised by the curvature of the $Q$-function estimate through the Legendre differential operator, and establish an $\mathcal{O}\!\left(N^{-\frac{1}{5}}\right)$ best-iterate finite-time convergence rate up to actor and critic approximation errors. Moreover, the mirror-descent step size $λ$ directly controls the target drift and hence actor tracking error, whereas the analogous Gibbs bound contains a non-vanishing tracking term.