Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces PremiseCheck, a novel framework for efficiently fact-checking peer review comments, addressing the growing challenge of review quality in academic publishing. PremiseCheck combines a large language model (LLM) for premise extraction with a decision model for fact verification, achieving accuracy comparable to frontier LLMs at significantly reduced cost and latency. The benchmark evaluates performance across NLP, machine learning, and natural sciences, demonstrating PremiseCheck's effectiveness in identifying factual errors in both human and AI-generated reviews.
A new framework slashes LLM fact-checking costs for peer reviews by 75% while maintaining state-of-the-art accuracy.
Peer review is under strain from the growth of submissions, and review comments written by humans or AI can misstate what a paper says. We introduce ReviewFact, a benchmark that labels such factual errors in review comments under one definition across NLP, machine learning, and the natural sciences, and that measures cost and latency alongside accuracy. A frontier LLM checks comments well but must read the whole paper for each check, while cheap decision models, which score whether a statement holds in a text, fall far short on whole comments that mix several implicit assertions. PremiseCheck is a framework that combines the two. An LLM rewrites each comment as explicit premises without reading the paper, and a decision model checks each premise against the paper. When each comment is checked on its own, PremiseCheck reaches accuracy comparable to GPT-5.5 across all three fields at about half its cost, and on NLP reviews at a quarter of its cost and less than half its latency.