Search papers, labs, and topics across Lattice.
This paper introduces Cooperative yet Critical reward modeling (C2), a framework that trains a reward model to critically collaborate with a rubric generator using only binary preferences. C2 synthesizes helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference, training a cooperative rubric generator and a critical verifier. Experiments show C2 outperforms reasoning reward models, achieving up to 6.5 points improvement on RM-Bench and matching the performance of a 4x larger model trained with external rubric annotations.
Reward models can achieve state-of-the-art performance by critically collaborating with a rubric generator trained solely from binary preferences, eliminating the need for costly rubric annotations.
Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find that rubric generation is vulnerable to a failure of cooperation; low-quality rubrics actively mislead reward models rather than help. Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences. In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4$\times$ larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makes reward models more trustworthy in a scalable way.
