Search papers, labs, and topics across Lattice.
This paper analyzes the convergence properties of Wasserstein Policy Optimization (WPO) in continuous state and action spaces, a gap in the existing literature. It leverages mean-field analysis and log-Sobolev inequalities to demonstrate that WPO within entropy-regularized Markov Decision Processes converges linearly. The analysis assumes a sufficiently regular solution to the gradient flow equation, enabling the proof of monotonic energy dissipation and the establishment of a local log-Sobolev inequality.
WPO, a promising RL algorithm for continuous control, is now proven to converge linearly, finally putting it on solid theoretical footing.
Wasserstein Policy Optimization (WPO) is a recently proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize stochastic policies in continuous action spaces. Despite its empirical success, the theoretical convergence properties of WPO in environments with continuous state and action spaces have yet to be fully established. In this note, we argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly. This is done by leveraging recent advances in mean-field analysis for convergence of gradient flows using log-Sobole inequalities. Assuming existence of sufficiently regular solution to the gradient flow equation we demonstrate monotonic energy dissipation along the flow and establish a local log-Sobolev inequality. Ultimately, these properties allow us to argue that the value function should converge linearly to the global optimum.
