Search papers, labs, and topics across Lattice.
This paper introduces StepPO, a novel approach to Agentic Reinforcement Learning (RL) that shifts the focus from token-level to step-level Markov Decision Processes (MDPs) for optimizing the decision-making capabilities of Large Language Models (LLMs). By advocating for step-level credit assignment, the authors address the challenges of delayed and sparse rewards in multi-turn interactive settings, which are critical for enhancing agentic capabilities such as tool use. Preliminary experiments suggest that this step-aligned paradigm significantly improves the alignment of policy optimization with the actual decision-making processes of LLM agents.
Shifting from token-level to step-level optimization could redefine how we train LLMs for complex, multi-turn interactions.
General agents have given rise to phenomenal applications such as OpenClaw and Claude Code. As these agent systems (a.k.a. Harnesses) strive for bolder goals, they demand increasingly stronger agentic capabilities from foundation Large Language Models (LLMs). Agentic Reinforcement Learning (RL) is emerging as a central post-training paradigm for empowering LLMs with these capabilities and is playing an increasingly pivotal role in agent training. Unlike single-turn token-level alignment or reasoning enhancement, as in RLHF and RLVR, Agentic RL targets multi-turn interactive settings, where the goal is to optimize core agentic capabilities such as decision making and tool use while addressing new challenges including delayed and sparse rewards, as well as long and variable context. As a result, the token-centric modeling and optimization paradigm inherited from traditional LLM RL is becoming increasingly inadequate for capturing real LLM agent behavior. In this paper, we present StepPO as a position on step-level Agentic RL. We argue that the conventional token-level Markov Decision Process (MDP) should be advanced to a step-level MDP formulation, and that the step, rather than the token, should be regarded as the proper action representation for LLM agents. We then propose step-level credit assignment as the natural optimization counterpart of this formulation, thereby aligning policy optimization and reward propagation with the granularity of agent decisions. Finally, we discuss the key systems designs required to realize step-level Agentic RL in practice and preliminary experiments provide initial evidence for the effectiveness of this perspective. We hope that the step-aligned, step-level paradigm embodied in StepPO offers the Agentic RL community a useful lens for understanding agent behavior and helps advance LLMs toward stronger general-agent capabilities.
