Search papers, labs, and topics across Lattice.
RUBRIC-ARROW addresses the challenge of pointwise reward modeling in non-verifiable domains by alternating training between a rubric generator and a rubric-conditioned judge. This framework uses a probability-based scoring rule to mitigate ties and phase-specific preference-based rewards to train the pointwise evaluator. Experiments demonstrate that RUBRIC-ARROW achieves competitive reward modeling accuracy and improves downstream policy post-training.
Stop struggling with subjective LLM evaluation: RUBRIC-ARROW aligns models better by alternating between generating evaluation rubrics and judging against them, using only pairwise preferences.
Pointwise reward modeling offers critical signals for LLM post-training, yet struggles with absolute scoring in subjective, non-verifiable settings. Rubric-based methods address this by decomposing evaluation into explicit criteria, but existing approaches typically depend on frontier LLMs and suffer from ties caused by hard Boolean aggregation. We present RUBRIC-ARROW, an alternating framework that jointly trains a rubric generator and a rubric-conditioned judge, with its RL stage using only pairwise preference data. Our method couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and an alternating GRPO scheme that together train the pointwise evaluator. Extensive experiments show that RUBRIC-ARROW achieves competitive reward-modeling accuracy and yields consistent gains for downstream policy post-training.
Microsoft Research