Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of smaller language models as judges in rubric-based reinforcement learning, addressing the high computational costs associated with using larger models. By constructing two evaluation datasets, PointRubric and RaR-Science-Static, the authors compare various methods for extracting judgments from small models, revealing that the Qwen3-1.7B Probe judge outperforms larger generative models in criterion-level agreement and efficiency. Notably, the Probe judge achieves a significant improvement in training policy rewards while requiring substantially less judging time compared to an 8B generative model baseline.
Smaller language models can efficiently replace larger ones in rubric-based reinforcement learning, achieving competitive performance with significantly reduced computational costs.
Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7$\times$ more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.
