Search papers, labs, and topics across Lattice.
This paper addresses the challenge of scaling peer review in the face of rapidly increasing AI-assisted scientific output by introducing the Paper Assistant Tool (PAT), an AI framework designed for comprehensive scientific evaluation. PAT utilizes inference scaling techniques to enhance its error detection capabilities, achieving a 34% improvement in identifying mathematical errors compared to traditional methods. Pilot deployments at major conferences show that PAT effectively identifies critical errors and suggests improvements, thereby reducing the cognitive load on human referees while maintaining their oversight in the review process.
A new AI tool can catch 34% more mathematical errors in scientific papers, transforming the peer review landscape.
Artificial intelligence is driving a revolution in scientific discovery, accelerating everything from hypothesis generation to mathematical theorem proving. However, this rapid acceleration is creating a systemic challenge: traditional human peer review cannot scale to match the influx of AI-assisted science. Ultimately, to resolve this tension, we must also deploy AI to accelerate the verification and review process itself. To frame the discussion around this transition, we propose a taxonomy consisting of four progressive levels of AI-human collaboration in scientific evaluation, and discuss various trade-offs involved with each. As a step toward this future, we introduce the Paper Assistant Tool (PAT), an agentic AI framework built for deep scientific review and verification. PAT ingests full scientific manuscripts and produces a comprehensive evaluation, checking theoretical results, validating experiments, suggesting improvements, and identifying potential flaws. By utilizing inference scaling techniques, PAT is able to identify deeper issues than a single model call alone, achieving a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. Pilot deployments of PAT as a pre-submission tool for authors at two major Computer Science conferences -- STOC and ICML -- demonstrate its ability to identify critical errors and suggest substantive improvements to research papers. By catching errors early, PAT eases the cognitive burden placed on referees, while preserving their control over the outcomes of the review process.
CMU ML
Google Research