Search papers, labs, and topics across Lattice.
This paper addresses the challenge of sparse data coverage in offline MARL by introducing PLCQL, a framework that learns a state-dependent partial action replacement (PAR) policy. PLCQL formulates PAR subset selection as a contextual bandit problem, using PPO with uncertainty-weighted rewards to dynamically determine the number of agents to replace. Experiments across MPE, MaMuJoCo, and SMAC benchmarks demonstrate that PLCQL achieves higher normalized scores on a majority of tasks and outperforms SPaCQL with significantly reduced computational cost.
Overcome the curse of dimensionality in offline MARL by learning which agents' actions to replace, achieving state-of-the-art performance with dramatically reduced computation.
Offline multi-agent reinforcement learning (MARL) faces a critical challenge: the joint action space grows exponentially with the number of agents, making dataset coverage exponentially sparse and out-of-distribution (OOD) joint actions unavoidable. Partial Action Replacement (PAR) mitigates this by anchoring a subset of agents to dataset actions, but existing approach relies on enumerating multiple subset configurations at high computational cost and cannot adapt to varying states. We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem and learns a state-dependent PAR policy using Proximal Policy Optimisation with an uncertainty-weighted reward. This adaptive policy dynamically determines how many agents to replace at each update step, balancing policy improvement against conservative value estimation. We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents. Compared with the previous PAR-based method SPaCQL, PLCQL reduces the number of per-iteration Q-function evaluations from n to 1, significantly improving computational efficiency. Empirically, PLCQL achieves the highest normalised scores on 66% of tasks across MPE, MaMuJoCo, and SMAC benchmarks, outperforming SPaCQL on 84% of tasks while substantially reducing computational cost.
