Search papers, labs, and topics across Lattice.
MADDPG-K is introduced to address the scalability issues of centralized critics in multi-agent reinforcement learning by restricting each agent's critic to information from its $k$ nearest neighbors. This approach maintains a constant-size critic input, independent of the total number of agents, thereby reducing computational costs. Empirical evaluations in cooperative and adversarial environments demonstrate that MADDPG-K achieves competitive or superior performance compared to standard MADDPG, with faster convergence and better runtime scaling.
MADDPG-K scales multi-agent learning by ditching the all-seeing critic for a neighborhood watch, achieving faster training and better performance without the quadratic cost of full observation.
We propose MADDPG-K, a scalable extension to Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that addresses the computational limitations of centralized critic approaches. Centralized critics, which condition on the observations and actions of all agents, have demonstrated significant performance gains in cooperative and competitive multi-agent settings. However, their critic networks grow linearly in input size with the number of agents, making them increasingly expensive to train at scale. MADDPG-K mitigates this by restricting each agent's critic to the $k$ closest agents under a chosen metric which in our case is Euclidean distance. This ensures a constant-size critic input regardless of the total agent count. We analyze the complexity of this approach, showing that the quadratic cost it retains arises from cheap scalar distance computations rather than the expensive neural network matrix multiplications that bottleneck standard MADDPG. We validate our method empirically across cooperative and adversarial environments from the Multi-Particle Environment suite, demonstrating competitive or superior performance compared to MADDPG, faster convergence in cooperative settings, and better runtime scaling as the number of agents grows. Our code is available at https://github.com/TimGop/MADDPG-K .
