Search papers, labs, and topics across Lattice.
This paper addresses the high variance issues in the joint advantage function during sequential updates in cooperative multi-agent reinforcement learning (MARL) by introducing a clipping objective that stabilizes advantage fluctuations. The authors provide both empirical and theoretical analyses demonstrating that their method achieves sub-linear convergence to 蔚-Nash Equilibria while maintaining low variance in advantage estimation. Experimental results across three benchmark environments indicate that the proposed approach consistently outperforms existing baselines, highlighting its effectiveness in enhancing stability and convergence in MARL settings.
Clipping advantage fluctuations can lead to stable convergence in cooperative multi-agent reinforcement learning, outperforming traditional methods.
Cooperative multi-agent reinforcement learning assumes each agent shares the same reward function and can be trained effectively using the Trust Region framework of single-agent. Instead of relying on other agents' actions, the independent actors setting considers each agent to act based only on its local information, thus having more flexible applications. However, in the sequential update framework, it is required to re-estimate the joint advantage function after each individual agent's policy step. Despite the practical success of importance sampling, the updated advantage function suffers from exponentially high variance problems, which likely result in unstable convergence. In this work, we first analyze the high variance advantage both empirically and theoretically. To overcome this limitation, we introduce a clipping objective to control the upper bounds of the advantage fluctuation in sequential updates. With the proposed objective, we provide a monotonic bound with sub-linear convergence to $蔚$-Nash Equilibria. We further derive two new practical algorithms using our clipping objective. The experiment results on three popular multi-agent reinforcement learning benchmarks show that our proposed method outperforms the tested baselines in most environments. By carefully analyzing different training settings, our proposed method is highlighted with both stable convergence properties and the desired low advantage variance estimation. For reproducibility purposes, our source code is publicly available at https://github.com/giangbang/Low-Variance-Trust-Region-MARL.
