Search papers, labs, and topics across Lattice.
This study evaluates the performance of emerging agentic review systems, specifically OpenAIReview, coarse, and Reviewer3, against traditional peer review benchmarks using six different LLMs. The findings reveal that OpenAIReview combined with GPT-5.5 achieves an impressive 83.0% accuracy in predicting paper quality based on external signals, while also detecting 71.6% of injected errors in a perturbation benchmark. Furthermore, user feedback from a public deployment indicates a favorable reception, highlighting the potential of AI-assisted reviews despite existing limitations.
OpenAIReview + GPT-5.5 not only predicts paper quality with 83% accuracy but also detects over 71% of critical errors, showcasing the promise of AI in peer review.
A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated. We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline, across six LLMs spanning frontier and efficient models. First, we study whether AI reviews on ICLR/NeurIPS papers track with papers' quality as approximated by external signals such as citations and acceptance decisions. Every system performs above chance in pairwise accuracy, and the best is OpenAIReview + GPT-5.5 at 83.0%. Second, to test whether systems can catch errors with known ground truth, we construct a perturbation benchmark that injects four categories of errors into papers across eight arXiv subject classes and measure detection recall. The strongest configuration (OpenAIReview + GPT-5.5) catches 71.6% of injected errors, leaving substantial room for improvement. The union of detections across six models reaches 83.3% recall, suggesting different models detect different errors and better harness design can potentially increase performance. Beyond these benchmarks, we study a public deployment of OpenAIReview with real users. Votes on its comments skew positive at 1.44 to 1, and the most common complaints are about false positives and minor nitpicks. Together, by evaluating full review systems backed by state-of-the-art models on real research papers, we show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.
