Search papers, labs, and topics across Lattice.
JudgmentBench, a new benchmark of 30 legal tasks, was created with both rubric scores and pairwise preference judgments from practicing attorneys to compare evaluation methodologies. The study found that pairwise preference judgments align much better with intended quality orderings than rubric-based scoring (Spearman's rho of 0.908 vs. 0.150). This suggests that comparative judgment is a more effective and efficient method for evaluating LLM outputs in high-expertise domains lacking ground truth.
Rubric-based scoring, a common evaluation method, is surprisingly bad at capturing quality in expert domains, with pairwise comparisons proving far more reliable and efficient.
Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts on the same items. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.
Stanford HAI