Search papers, labs, and topics across Lattice.
The paper introduces SAVE, a self-supervised framework for improving reward models (RMs) by leveraging on-policy responses graded using the policy's value function as feedback. This approach converts reward-graded responses into supervision using a prompt-specific value head as an adaptive anchor and updates the RM via a contrastive objective based on RM advantages and filtered ambiguous samples. Empirical results across six benchmarks demonstrate that SAVE consistently outperforms existing methods while improving performance across various RL algorithms and policy backbones.
Forget expensive human preference data: this new method uses the policy's own value function to self-supervise reward model training, boosting performance across diverse benchmarks and RL algorithms.
Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.
