Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study evaluates the alignment of reviews generated by three large language models (LLMs) with human peer reviews and conference decisions for 300 ICLR 2026 submissions. While all LLMs effectively distinguished between accepted and rejected papers, they failed to replicate the nuanced oral versus poster distinction found in human evaluations. Notably, the models exhibited provider-specific scoring patterns and thematic differences in identified weaknesses, highlighting that broad alignment does not guarantee agreement on finer review criteria.
LLMs may accurately identify accepted papers but struggle with the nuanced distinctions that human reviewers prioritize, revealing critical gaps in their evaluative capabilities.
Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting. We compare reviews from OpenAI GPT-5.4, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.6 with human reviews and final decisions for 300 topic-matched ICLR 2026 submissions, equally divided among oral, poster, and rejected papers. Each model reviewed every paper using identical instructions and rating scales after decision information was removed. Our study contributes a cross-provider analysis of three complementary dimensions: alignment with broad and fine-grained decision categories, differences in recommendation-scale usage, and thematic agreement in identified weaknesses. All three LLMs distinguished accepted from rejected papers, but none reproduced the oral versus poster distinction present in human ratings. Scoring patterns were provider-specific: Gemini assigned systematically higher ratings, while OpenAI and Claude were closer to humans for rejected and poster papers but more critical of oral papers. Human and LLM reviews also differed in emphasis, with LLMs more frequently identifying missing baseline comparisons and humans more often raising computational-efficiency concerns. These results show that broad decision alignment does not imply agreement with finer human judgments or reviewing priorities.