Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces StepPO, a novel step-aligned policy optimization framework designed to enhance agentic reinforcement learning (RL) for large language models (LLMs). By reformulating the traditional token-centric Markov Decision Process (MDP) into a step-level MDP, StepPO aligns policy optimization with the natural decision-making granularity of LLM agents, allowing for more effective multi-turn interactions. Experimental results demonstrate that StepPO outperforms existing RL algorithms across various tasks, highlighting its potential to improve LLM agent capabilities significantly.
StepPO reveals that aligning policy optimization with agent decision-making steps can lead to superior performance in multi-turn interactions, outperforming traditional RL methods.
Agentic reinforcement learning (RL) is emerging as a critical post-training paradigm for improving LLM agent capabilities. Existing RL algorithms for LLMs largely follow the token-centric paradigm as in RLHF and RLVR, where tokens serve as the basic units for modeling and optimization. However, this paradigm introduces a granularity mismatch in agentic RL, as it optimizes token-level predictions while LLM agents make step-level decisions through cycles of environmental observations and actions. To bridge this gap, we propose StepPO, a step-centric paradigm for agentic RL via step-aligned policy optimization. Specifically, we reformulate agentic RL from a token-level Markov Decision Process (MDP) into a step-level MDP, where interaction steps serve as the basic trajectory representations. We further propose step-level credit assignment to align policy optimization with the natural granularity of agent decisions. Together, StepPO optimizes agent policies at the step level for multi-turn agent-environment interaction. Experiments across multi-hop QA, academic paper search, and text-world action tasks show that StepPO consistently outperforms various RL algorithms. Further analyses provide insights into how step-centric paradigm improves agent training. We hope this step-centric paradigm offers a useful lens for understanding agent behavior and a practical path for training more capable LLM agents.