Search papers, labs, and topics across Lattice.
This paper introduces Ranking-based Reward Construction (RRC), a novel approach that enhances the integration of generative reward models in reinforcement learning (RL) by aligning their comparative nature with the scalar scoring paradigm of existing RL algorithms. By employing self-competitive ranking and anchor-guided ranking strategies, RRC effectively derives rewards from relative preference rankings, leading to improved learning signals for RL. Experimental results show that RRC significantly outperforms traditional reward construction methods across various benchmarks in open-ended chat and reasoning tasks.
Generative reward models can finally unlock their full potential in RL, leading to substantial performance improvements through innovative ranking strategies.
Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.
