Search papers, labs, and topics across Lattice.
This paper introduces VIMPO, a novel critic-free policy optimization method for large language models that leverages a policy-implied value function derived from KL-regularized reinforcement learning. By avoiding the instability associated with learned value functions, VIMPO effectively incorporates outcome-level verifiable rewards while maintaining simplicity in training. Experimental results show that VIMPO outperforms existing group-relative methods like GRPO across multiple mathematical benchmarks, particularly excelling in competition-style evaluations and demonstrating robustness under noisy reward conditions.
VIMPO achieves superior performance in reasoning tasks by enabling fine-grained credit assignment without the complexities of a critic, redefining the landscape of reinforcement learning for LLMs.
Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment. Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token. Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability. We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from the optimality conditions of KL-regularized reinforcement learning. For autoregressive generation, the resulting value recurrence can be written in terms of policy-reference log-ratios and anchored by the terminal condition that no future reward remains at the end of a trajectory. This gives a simple value loss that incorporates outcome-level verifiable rewards without training a critic. The same derivation also yields a critic-free actor advantage, allowing VIMPO to separate reward incorporation through the value loss from policy improvement through a PPO-style actor update. On mathematical RLVR benchmarks, VIMPO improves over GRPO across MATH-500, AIME 2024, AIME 2025, and OlympiadBench, with especially larger gains on competition-style evaluations. Under noisy rewards, VIMPO retains a consistent advantage over GRPO, suggesting that policy-implied value optimization can provide finer credit assignment while preserving the practical simplicity of critic-free training.
BAIR