Search papers, labs, and topics across Lattice.
This paper introduces a novel approach, Conditional Proximal Policy Optimization (CPPO), to train diffusion policies within an on-policy reinforcement learning framework. CPPO overcomes the challenge of computing action log-likelihoods in diffusion models by aligning policy iteration with the diffusion process, requiring only the evaluation of a simple Gaussian probability. The method demonstrates superior performance and multimodal policy behavior on benchmark tasks in IsaacLab and MuJoCo Playground, while also naturally handling entropy regularization.
On-policy reinforcement learning with diffusion policies just got a whole lot easier thanks to a clever method that sidesteps expensive log-likelihood computations.
Reinforcement learning (RL) has been extensively employed in a wide range of decision-making problems, such as games and robotics. Recently, diffusion policies have shown strong potential in modeling multi-modal behaviors, enabling more diverse and flexible action generation compared to the conventional Gaussian policy. Despite various attempts to combine RL with diffusion, a key challenge is the difficulty of computing action log-likelihood under the diffusion model. This greatly hinders the direct application of diffusion policies in on-policy reinforcement learning. Most existing methods calculate or approximate the log-likelihood through the entire denoising process in the diffusion model, which can be memory- and computationally inefficient. To overcome this challenge, we propose a novel and efficient method to train a diffusion policy in an on-policy setting that requires only evaluating a simple Gaussian probability. This is achieved by aligning the policy iteration with the diffusion process, which is a distinct paradigm compared to previous work. Moreover, our formulation can naturally handle entropy regularization, which is often difficult to incorporate into diffusion policies. Experiments demonstrate that the proposed method produces multimodal policy behaviors and achieves superior performance on a variety of benchmark tasks in both IsaacLab and MuJoCo Playground.
