Search papers, labs, and topics across Lattice.
This study systematically investigates the regulatory landscape of AI-assisted peer review across 111 AI/NLP conferences and medical journals, revealing significant differences in policies between these communities. Additionally, it evaluates the quality of AI-generated peer reviews from ICLR 2026 and Nature Communications, utilizing a novel dataset to compare human and machine-generated reviews across various metrics. The findings indicate that while current LLMs produce fluent reviews, they tend to provide overly positive recommendations and lack depth in critical analysis, highlighting the need for multi-dimensional evaluation metrics to accurately assess review quality.
Current LLMs may sound fluent, but they often deliver overly positive and generic peer reviews that misrepresent their true quality.
AI-assisted peer review is increasingly discussed and adopted as a tool to support the scientific publishing process, yet there is little systematic understanding of how publication venues regulate its use or of how capable current AI review systems are. We address these questions by first surveying reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals, revealing substantial regulation differences between the two communities. Second, we evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset comprising original manuscript submissions and several hundred human- and machine-generated reviews. We compare reviews produced by open-source and proprietary models using complementary evaluation metrics, including LLM-as-a-Judge, score alignment, granularity, and overlap with human reviewers'concerns. Our results show that current LLMs can generate detailed and fluent reviews but exhibit systematic weaknesses, such as overly positive recommendations, generic criticism, and uneven evidence grounding. We demonstrate that aggregate quality scores alone can overestimate review quality and argue for multi-dimensional evaluation of AI-generated peer reviews.
