Search papers, labs, and topics across Lattice.
This paper introduces Graph-based Group Policy Optimization (GraphGPO), a novel credit assignment method for agentic RL that moves beyond trajectory-level attribution. GraphGPO constructs a state-transition graph from all rollout trajectories and estimates the distance to the goal from each state using global graph information. By assigning credit based on a graph-based advantage function that measures the reduction in distance to the goal, GraphGPO achieves state-of-the-art performance and improved training efficiency on challenging benchmarks.
Don't let valuable steps in failed trajectories go unnoticed: GraphGPO leverages state-transition graphs for fine-grained credit assignment in agentic RL, boosting performance and efficiency.
Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs) and have been rapidly extended to agentic tasks. However, their credit assignment relies heavily on coarse-grained trajectory-level attribution according to final outcomes, making it difficult to capture the contribution of individual steps, such as valuable steps obscured within failed trajectories. To uncover latent information and enable more faithful step-level credit assignment, we propose Graph-based Group Policy Optimization (GraphGPO), which first aggregates all rollout trajectories into a unified state-transition graph and then estimates the distance from each state to the task goal using the global information encoded in the graph. Finally, GraphGPO assigns credit to each edge by estimating a graph-based advantage, based on how much the transition reduces the distance to the task goal. In this way, GraphGPO significantly improves training efficiency and achieves state-of-the-art performance across a range of challenging benchmarks.
