Search papers, labs, and topics across Lattice.
This paper addresses the challenges of policy transfer in cooperative multi-agent reinforcement learning (MARL) when objectives change dynamically post-deployment. It highlights that independent policy composition can lead to suboptimal joint behaviors, a phenomenon absent in single-agent scenarios, and introduces MA-USFA, a hierarchical method that ensures safe and flexible policy composition across agents. The key finding is that synchronized composition is insufficient for diverse objectives, while MA-USFA effectively predicts successor features conditioned on teammates' objectives, maintaining performance across varying tasks without the need for retraining.
Independent policy composition in multi-agent systems can lead to worse outcomes than any individual policy in the library, challenging conventional wisdom in reinforcement learning.
Many reinforcement learning systems, from fleet management to traffic signal control, must serve an objective that changes dynamically after deployment, and retraining a policy for each new objective is prohibitively expensive. For a single agent, this problem is well understood: successor features with generalized policy improvement, together with their universal extension, recombine a library of learned policies into a policy for any new objective, with a guarantee that the result is never worse than any policy in the library. However, multi-agent transfer has received far less attention, and the common practice of letting each agent recombine its own library independently inherits the recipe but not the guarantee. We prove that this independent composition can produce joint behavior strictly worse than every policy in the library, because recombining teammates changes the environment each agent faces and invalidates the values it relies on, a failure with no single-agent counterpart. We further show that the only unconditionally safe fixed rule is synchronized composition, which moves the whole team to one jointly trained policy but cannot serve objectives that assign different goals to different agents. To attain safety and flexibility at once, we propose MA-USFA, a hierarchical method with two layers: a lower layer of universal successor feature approximators that predicts each agent's successor features while conditioned on its teammates'objectives, and an upper composer that selects, across agents, which library entry each agent should follow and supplies the cross-agent correction a per-agent value cannot represent. Trained once over the distribution of objectives, it is applied at deployment with no per-task adaptation.
