Search papers, labs, and topics across Lattice.
The paper addresses the challenge of scaling cooperative multi-agent reinforcement learning (MARL) due to cross-agent noise, where gradient variance scales linearly with the number of agents. They introduce Descent-Guided Policy Gradient (DG-PG), which leverages differentiable analytical models of the environment to construct noise-free per-agent guidance gradients, effectively decoupling each agent's gradient from the actions of others. The authors demonstrate that DG-PG reduces gradient variance to O(1), achieves agent-independent sample complexity, and converges in a cloud scheduling task with up to 200 agents, while baselines fail.
Forget scaling bottlenecks: DG-PG uses differentiable analytical models to slash gradient variance in cooperative MARL, achieving convergence in a 200-agent cloud scheduling task where standard methods fail.
Scaling cooperative multi-agent reinforcement learning (MARL) is fundamentally limited by cross-agent noise: when agents share a common reward, the actions of all $N$ agents jointly determine each agent's learning signal, so cross-agent noise grows with $N$. In the policy gradient setting, per-agent gradient estimate variance scales as $Θ(N)$, yielding sample complexity $\mathcal{O}(N/ε)$. We observe that many domains -- cloud computing, transportation, power systems -- have differentiable analytical models that prescribe efficient system states. In this work, we propose Descent-Guided Policy Gradient (DG-PG), a framework that constructs noise-free per-agent guidance gradients from these analytical models, decoupling each agent's gradient from the actions of all others. We prove that DG-PG reduces gradient variance from $Θ(N)$ to $\mathcal{O}(1)$, preserves the equilibria of the cooperative game, and achieves agent-independent sample complexity $\mathcal{O}(1/ε)$. On a heterogeneous cloud scheduling task with up to 200 agents, DG-PG converges within 10 episodes at every tested scale -- from $N=5$ to $N=200$ -- directly confirming the predicted scale-invariant complexity, while MAPPO and IPPO fail to converge under identical architectures.
Tsinghua AI