Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of various off-the-shelf LLM judges in assessing citation quality for deep-research systems, focusing on two key dimensions: source relevance and factual support. By benchmarking eight models across 1,248 human-reviewed decisions, the authors reveal that less expensive models can perform competitively, with GPT-5-mini achieving the highest source-relevance score. However, significant differences in pass-rate drift and error rates among the judges indicate that calibration is essential before using these models as reliable reward signals in reinforcement learning contexts.
Cheaper LLM judges can match the performance of their more expensive counterparts in citation verification, challenging the assumption that only high-cost models are suitable for deep-research tasks.
Reinforcement learning increasingly relies on an LLM judge to score each rubric criterion, and that judge acts as the reward model during training. Before such a signal can be trusted, we need to know how capable the judge must be and how biased it is. We study this calibration question for citation quality in deep-research systems, where a search-grounded LLM must support each claim it writes with a cited source. Citation quality is a structured rubric task in which each attribution-citation pair is judged along two dimensions that require an LLM, source relevance and factual support. On an adversarial long-form benchmark, we score 8 off-the-shelf LLM judges from 3 model families against gold labels over 1,248 rubric decisions, all of which were human-reviewed and 378 of which were hard cases adjudicated from judge disagreements. Cheaper judges remain competitive across both dimensions, with GPT-5-mini attaining the strongest source-relevance pass-class F1 at 0.908 ($\kappa$=0.636), while on factual support the judges are statistically indistinguishable (overlapping confidence intervals), so no single model dominates. At comparable F1, the judges still differ substantially in pass-rate drift, false positive rate, and false negative rate. Scalar F1 obscures this directional bias, yet it is exactly what a downstream reinforcement learning loop would reinforce. Calibrating the judge is therefore a prerequisite for using citation rubrics as reward signals, and our results show that this calibration does not require the most expensive available model.
