Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
Addressing the acute instability of token-level credit assignment under sparse rewards, the authors formulate $\pi$PPO, an actor-critic RLVR framework that grants the critic self-privileged access to contrastive same-prompt rollouts. Standard state-only critics typically fail because they must simultaneously predict intermediate task progress and anticipate non-stationary policy dynamics from an isolated partial trajectory. Across competitive mathematical reasoning benchmarks, grounding the critic with both successful and failed rollout "foils" substantially improves value estimation accuracy and downstream policy performance over standard PPO and critic-free baselines, even when deploying downscaled asymmetric critics.
Exposing the critic to both successful and failed same-prompt rollouts as contrastive foils resolves PPO's notoriously brittle credit assignment in multi-step reasoning—retaining its efficacy even when paired with a downscaled critic.
Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $π$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $π$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $π$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.