Search papers, labs, and topics across Lattice.
Graph-GRPO is introduced to address the challenges of high gradient variance and credit assignment in reinforcement learning-based topology optimization for LLM-based multi-agent systems. It samples a group of diverse communication graphs for each query and computes the advantage of specific edges based on their relative performance within the group, normalizing rewards to mitigate noise from task difficulty. Experiments on reasoning and code generation benchmarks show that Graph-GRPO outperforms state-of-the-art baselines in training stability and identifying critical communication pathways.
By normalizing rewards across groups of sampled communication graphs, Graph-GRPO stabilizes multi-agent topology learning and uncovers critical communication pathways obscured by noisy, absolute rewards.
Optimizing communication topology is fundamental to the efficiency and effectiveness of Large Language Model (LLM)-based Multi-Agent Systems (MAS). While recent approaches utilize reinforcement learning to dynamically construct task-specific graphs, they typically rely on single-sample policy gradients with absolute rewards (e.g., binary correctness). This paradigm suffers from severe gradient variance and the credit assignment problem: simple queries yield non-informative positive rewards for suboptimal structures, while difficult queries often result in failures that provide no learning signal. To address these challenges, we propose Graph-GRPO, a novel topology optimization framework that integrates Group Relative Policy Optimization. Instead of evaluating a single topology in isolation, Graph-GRPO samples a group of diverse communication graphs for each query and computes the advantage of specific edges based on their relative performance within the group. By normalizing rewards across the sampled group, our method effectively mitigates the noise derived from task difficulty variance and enables fine-grained credit assignment. Extensive experiments on reasoning and code generation benchmarks demonstrate that Graph-GRPO significantly outperforms state-of-the-art baselines, achieving superior training stability and identifying critical communication pathways previously obscured by reward noise.
Tsinghua AI