Search papers, labs, and topics across Lattice.
This paper introduces HalluPeer, a novel benchmark designed specifically to detect hallucinations in scientific peer reviews, addressing the limitations of existing benchmarks that do not account for the complexities of verifying claims in lengthy technical documents. By creating a taxonomy of hallucinations and providing annotated triples of paper content, human reviews, and hallucination-injected reviews, the authors demonstrate that current detection methods are inadequate in distinguishing between valid critiques and hallucinations. Experiments involving 12,000 papers and 38,000 reviews reveal that HalluPeer's defined patterns of hallucinations are prevalent in real peer reviews, underscoring the necessity for improved verification mechanisms in academic publishing.
Existing hallucination detection methods falter in the nuanced landscape of scientific peer reviews, revealing a critical gap in ensuring review reliability.
The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but unsupported claims that undermine review reliability. Existing hallucination benchmarks are not designed for peer review, where verification requires grounding claims in long, technical papers. We introduce HalluPeer, a benchmark for detecting hallucinations in scientific peer reviews, providing aligned triples of paper content, human-written reviews, and hallucination-injected reviews, annotated for detection, classification, and localization. Our pipeline induces a peer-review-specific hallucination taxonomy, identifies review contexts, and injects hallucinations with automated filtering. Experiments on 12K papers and 38K reviews show that existing detectors struggle to separate hallucinations from legitimate critique, while evaluation on authentic reviews demonstrates that HalluPeer-defined hallucination patterns occur in real peer reviews, highlighting the critical need for source-aware verification. Our project page can be found in https://github.com/Lin-TzuLing/HalluPeer.git
