Search papers, labs, and topics across Lattice.
This paper introduces Agent-Chained Policy Optimization (ACPO), a novel approach for Multi-Agent Reinforcement Learning (MARL) that enables decentralized training while maintaining joint policy gradient optimization. By leveraging a decentralized decomposition of per-agent terms and a belief-based coordination mechanism, ACPO allows agents to update independently yet effectively contribute to a collective policy improvement. Experimental results demonstrate that ACPO significantly outperforms existing methods, particularly as the number of agents increases, highlighting its scalability and effectiveness in cooperative tasks.
ACPO achieves joint policy optimization in MARL by enabling independent agent updates that effectively coordinate through a belief mechanism, outperforming traditional methods as agent numbers grow.
Cooperative tasks in Multi-Agent Reinforcement Learning (MARL) require agents to collectively maximize a shared return. Under the Centralized Training with Decentralized Execution (CTDE) paradigm, policy gradients have remained difficult to compute directly. Prior methods largely follow two approaches: independent factorized updates with centralized critics, which lack general joint-improvement guarantees without value decomposition assumptions, or alternating best-response updates, which can converge to suboptimal Nash Equilibria. In this paper, we show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics. Based on this decomposition, we develop Agent-Chained Policy Optimization (ACPO), where actors are trained independently, with their updates together constituting a single step on the joint policy gradient. Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions. The belief acts as the coordination mechanism which ties the independent per-agent updates into a joint gradient step. We evaluate ACPO on Multi-Robot Warehouse, SMACv2, and MA-MuJoCo, where it outperforms strong baselines, with the gap widening as the number of agents grows.
