Search papers, labs, and topics across Lattice.
This paper investigates the impact of inter-policy diversity on learning efficiency in ensemble policy gradient methods for large-scale reinforcement learning. They theoretically analyze the trade-offs between exploration breadth and quality, and propose Coupled Policy Optimization (CPO) to regulate diversity using KL constraints between policies within the ensemble. CPO outperforms strong baselines like SAPG, PBT, and PPO on dexterous manipulation tasks, demonstrating improved sample efficiency and final performance through structured exploration.
Taming policy diversity with KL constraints unlocks surprisingly stable and sample-efficient ensemble reinforcement learning in high-dimensional manipulation tasks.
Scaling reinforcement learning to tens of thousands of parallel environments requires overcoming the limited exploration capacity of a single policy. Ensemble-based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise training stability. In this work, we theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong baselines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation is key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at https://naoki04.github.io/paper-cpo/ .
