Search papers, labs, and topics across Lattice.
This paper introduces ALTERNATING-MARL, an alternating learning framework for cooperative Markov games with a global agent interacting with $n$ homogeneous local agents under communication constraints, where the global agent observes only a subset $k$ of local agent states. The global agent uses subsampled mean-field Q-learning against a fixed local policy, while local agents optimize in an induced MDP. The authors prove that this approach converges to an $\widetilde{O}(1/\sqrt{k})$-approximate Nash Equilibrium, achieving sample complexity separation between joint state and action spaces.
Subsampling agent states in cooperative MARL provably converges to approximate Nash equilibria, enabling efficient learning in communication-constrained multi-agent systems.
Many large-scale platforms and networked control systems have a centralized decision maker interacting with a massive population of agents under strict observability constraints. Motivated by such applications, we study a cooperative Markov game with a global agent and $n$ homogeneous local agents in a communication-constrained regime, where the global agent only observes a subset of $k$ local agent states per time step. We propose an alternating learning framework $(\texttt{ALTERNATING-MARL})$, where the global agent performs subsampled mean-field $Q$-learning against a fixed local policy, and local agents update by optimizing in an induced MDP. We prove that these approximate best-response dynamics converge to an $\widetilde{O}(1/\sqrt{k})$-approximate Nash Equilibrium, while yielding a separation in the sample complexities between the joint state space and action space. Finally, we validate our results in numerical simulations for multi-robot control and federated optimization.
