Search papers, labs, and topics across Lattice.
This paper introduces the concept of "progress advantage," an implicit advantage derived from reinforcement learning (RL) post-training that enables effective step-level scoring of LLMs without the need for dedicated reward models. By leveraging the log-probability ratio between the RL-trained policy and its reference policy, the authors demonstrate that this approach is annotation-free and domain-agnostic, providing significant advantages in various applications such as test-time scaling and uncertainty quantification. The results show that progress advantage consistently outperforms confidence-based baselines and even surpasses dedicated reward models across multiple benchmarks and model families.
Progress advantage reveals a powerful, annotation-free scoring mechanism that outperforms traditional reward models in LLM agentic settings.
Process reward models enable fine-grained, step-level evaluation of LLMs, yet building them for agentic settings remains prohibitively difficult: long-horizon interactions, irreversible actions, and stochastic environment feedback make both human annotation and Monte Carlo estimation infeasible at scale. In this work, we show that reinforcement learning (RL) post-training already provides the ingredients for effective step-level scoring, eliminating the need for dedicated reward model training altogether. Concretely, we derive an implicit advantage under a general stochastic Markov decision process, which we term progress advantage -- log-probability ratio between the RL-trained policy and its reference policy exactly recovers the optimal advantage function. This formulation makes the resulting signal annotation-free, domain-agnostic, and available as a byproduct of the standard RL post-training pipeline. We validate the effectiveness of the progress advantage across three different applications: test-time scaling, uncertainty quantification, and failure attribution on five benchmarks and four model families. Across all settings, it consistently outperforms confidence-based baselines and, despite requiring no task-specific training, surpasses dedicated trained reward models. We complement these results with deeper analyses on characteristics of progress advantage, offering practical guidance for adoption in real-world agentic systems.
