Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ReVeR, a novel pointwise rubric reward model that derives evaluation criteria directly from instructions, mitigating response-induced bias and enabling independent response assessment via tool-augmented judging. By training through label-blind distillation, ReVeR achieves superior preference accuracy over existing pairwise rubric RMs, particularly for instruction following, and demonstrates improved test-time scaling and downstream policy optimization. The study also reveals a crucial distinction between rubric fidelity, preference accuracy, and downstream utility, offering new directions for reward model research.
Evaluating LLM responses using criteria derived solely from instructions, not observed outputs, unlocks significantly better instruction following and scalability.
Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.