Search papers, labs, and topics across Lattice.
This paper introduces SPO++, an enhancement of Single-stream Policy Optimization (SPO) that addresses inefficiencies in group-relative reinforcement learning by standardizing terminal-outcome advantages based on action-token measures. By reorganizing prompt evidence according to the policy event that generated it, SPO++ achieves improved online learning efficiency in asynchronous agentic reinforcement learning tasks. The results demonstrate that SPO++ outperforms its predecessor, SPO, in both ALFWorld and Math-TIR environments, with action-token normalization identified as the most significant contributor to this improvement.
Standardizing terminal-outcome advantages can significantly boost online learning efficiency in asynchronous reinforcement learning scenarios.
Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.
