Search papers, labs, and topics across Lattice.
This paper introduces Single-rollout Autoregressive Policy Optimization (SAPO), a novel framework for agentic reinforcement learning that leverages a shared autoregressive backbone for policy and value function predictions. By addressing limitations of existing critic-free methods, SAPO achieves superior performance on long-horizon tasks while significantly reducing memory overhead and runtime costs. Experimental results demonstrate that SAPO outperforms traditional proximal policy optimization (PPO) and group-relative policy optimization (GRPO) by substantial margins, enhancing both stability and efficiency in training.
SAPO achieves a 15.1 percentage point improvement over PPO while slashing memory costs and runtime by a third, revolutionizing how we optimize agentic RL.
Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO.
