Search papers, labs, and topics across Lattice.
This paper introduces Group-Graph Policy Optimization (G2PO), a novel algorithm designed to enhance long-horizon agentic reinforcement learning by transforming linear interaction trajectories into a global state-transition graph. By employing group-aggregation state-value estimation and redefining agent actions as transitions between state nodes, G2PO addresses the issues of reward sparsity and coarse credit assignment that plague existing frameworks. Experimental results on long-horizon benchmarks show that G2PO significantly outperforms state-of-the-art methods, achieving success rate improvements of up to 22.2% over previous approaches.
G2PO redefines agent actions and leverages a global state-transition graph, leading to a 22.2% boost in success rates for long-horizon tasks.
Group-based Reinforcement Learning (RL) has significantly enhanced Large Language Models (LLMs) in agentic scenarios. To achieve finer-grained policy updates, recent agentic RL frameworks have shifted from trajectory-level to step-level training. However, long-horizon agentic RL suffers from severe reward sparsity and delay, as feedback is often deferred for dozens of interaction steps. While existing step-level frameworks refine training granularity, their credit assignment remains coarse-grained and still treats agent exploration as isolated, linear trajectories. This oversimplified perspective ignores the inherent graph structure of state transitions, leading to high-variance state-value estimation and myopic, localized credit assignment. To overcome these critical bottlenecks, we propose Group-Graph Policy Optimization (G2PO), a novel group-based RL algorithm tailored for multi-turn agentic tasks. G2PO explicitly transforms linear interaction trajectories into a global state-transition graph. By aggregating identical observations across different trajectories, we introduce group-aggregation state-value estimation that reduces sampling variance and trajectory-dependent bias. Furthermore, we redefine agent actions as transitions between state nodes and propose an edge-centric advantage estimation strategy. By globally standardizing Temporal Difference (TD) errors across the entire graph, G2PO explicitly identifies and prioritizes critical transitions that drive absolute task progress. Extensive experiments on representative long-horizon benchmarks-WebShop, ALFWorld, and AppWorld-demonstrate that G2PO substantially outperforms state-of-the-art prompt-based and RL baselines, achieving remarkable success rate improvements of up to 22.2% over GRPO.
Microsoft Research