Search papers, labs, and topics across Lattice.
This paper introduces Semantic Consistency Policy Optimization (SCPO), a novel approach to mitigate semantic credit inconsistency in group-based reinforcement learning for large language model (LLM) agents. By recovering step-level credit from successful siblings within the same rollout group, SCPO enhances the learning signal for intermediate steps in long-horizon, sparse-reward tasks. Experimental results on ALFWorld and WebShop demonstrate that SCPO achieves up to 93.7% success on ALFWorld and 74.8% on WebShop, significantly improving performance on challenging multi-step tasks compared to existing baselines.
SCPO transforms reinforcement learning for LLM agents by ensuring that intermediate steps receive consistent credit, leading to remarkable performance gains on complex tasks.
Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.
