Search papers, labs, and topics across Lattice.
This paper introduces "Beyond Rating," a new evaluation framework for AI peer reviewers that moves beyond simple rating prediction to assess the quality of textual justifications across five dimensions: Content Faithfulness, Argumentative Alignment, Focus Consistency, Question Constructiveness, and AI-Likelihood. They propose a Max-Recall strategy to handle expert disagreement and create a high-quality dataset of papers and reviews. Experiments show that text-centric metrics, especially recall of weakness arguments, correlate strongly with rating accuracy, highlighting the importance of aligning AI critique focus with human experts.
Current benchmarks miss the point: the real value of AI peer review lies in the quality of its textual justification, not just predicting a rating.
The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks that treat reviewing primarily as a rating prediction task. We argue that the utility of a review lies in its textual justification--its arguments, questions, and critique--rather than a scalar score. To address this, we introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions: Content Faithfulness, Argumentative Alignment, Focus Consistency, Question Constructiveness, and AI-Likelihood. Notably, we propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews, rigorously filtered to remove procedural noise. Extensive experiments demonstrate that while traditional n-gram metrics fail to reflect human preferences, our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy. These findings establish that aligning AI critique focus with human experts is a prerequisite for reliable automated scoring, offering a robust standard for future research.
