Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper introduces Continual Graph Multi-Agent Reinforcement Learning (CGMARL) to address the challenge of agents learning cooperative policies across task sequences with varying underlying structures, which existing methods fail to leverage for transfer and forgetting mitigation. The authors propose mapping task sequences to attributed graphs to model task-specific structures and introduce the Graph-based Formation (GRAFO) benchmark to highlight forgetting issues. A novel method, Frozen Graph Encoder (FROG), is presented, which utilizes a frozen graph backbone to retain past structural information, significantly improving performance when combined with existing continual learning techniques.
A frozen graph backbone effectively preserves past structural knowledge in continual multi-agent reinforcement learning, dramatically reducing forgetting and boosting transfer across tasks with distinct graph structures.
In Continual Multi-Agent Reinforcement Learning (CMARL), agents learn cooperative policies across sequences of tasks, aiming to adapt effectively to new tasks while preserving the ability to solve previously encountered ones. In many applications, tasks differ in their underlying structure, which can represent, for example, distinct operational conditions or target configurations (e.g., different network topologies in power grids or arrangements in formation control). Existing CMARL methods lack dedicated mechanisms to leverage this structural information when learning new tasks, failing to promote transfer and mitigate forgetting. To fill this gap, we propose Continual Graph Multi-Agent Reinforcement Learning (CGMARL), a novel framework for CMARL problems in which task sequences are mapped into a series of attributed graphs, each modeling a task-specific structure. In CGMARL, each graph determines the environment dynamics (next states and/or rewards) and the number of agents for the corresponding task. Then, we present Graph-based Formation (GRAFO), the first CGMARL benchmark, and show how forgetting arises in this setting. Finally, to address this limitation, we propose Frozen Graph Encoder (FROG), a method that relies on a frozen graph backbone to preserve past structural information in graph-based CMARL policies. Experiments on GRAFO show that pairing FROG with existing CL methods substantially improves performance on multiple CGMARL scenarios.