Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces the Peer Review AI Benchmark (PRAIB) to evaluate how well LLMs engage with scientific manuscripts compared to human reviewers, using metrics for review specificity, style, and behavior. They generated 11,000 reviews using five LLMs on 1,000 ICLR/NeurIPS papers and compared them to human reviews. Results show LLM reviews are less variable, positively biased, overconfident, and miss atomic weaknesses identified by humans, highlighting behavioral divergences.
LLMs may write longer, more complex peer reviews, but they consistently miss the critical, atomic weaknesses that human reviewers catch.
The growing number of submitted papers has motivated the exploration of Large Language Models (LLMs) as a means to support and augment the peer review process, particularly in terms of improving its speed and scalability. Yet, it remains unknown whether LLMs engage with scientific manuscripts in the same manner as human reviewers, or whether they merely produce review-looking text. To address this, we introduce the Peer Review AI Benchmark (PRAIB), a novel framework comprising thoroughly defined metrics that measure review specificity, style, and behavior of engagement. To complement the PRAIB framework, we conduct a large-scale empirical study leveraging a dataset of 11,000 reviews generated by five proprietary and open-source models for 1,000 ICLR and NeurIPS papers. Spanning the 2021--2025 period, these machine-generated reviews are compared against original human feedback across diverse prompting strategies to identify systematic behavioral divergences. Our analysis reveals that the generated reviews diverge significantly from feedback provided by human reviewers: LLM ratings are less variable, positively biased, and overconfident, and their cross-reference patterns are model-dependent and distinct from human norms. Furthermore, when evaluated through PRAIB, we observe that LLMs tend to generate longer, more complex reviews, yet frequently overlook the atomic weaknesses noted by human reviewers. By characterizing where and how LLMs reviewing behavior departs from human norms, PRAIB provides the community with a diagnostic tool for identifying which aspects of the review process LLMs can reliably support today and which require further development before deployment.