Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To mitigate citation drift and unsubstantiated claims in academic literature, the authors developed RefVerifier, an automated pipeline that extracts cited assertions from complete manuscripts and verifies them directly against resolved open-access source PDFs. This directly addresses the failure of existing RAG and fact-checking tools to handle end-to-end document context or bounded scholarly citation graphs. Evaluated on public benchmarks and full manuscripts, the system achieves an F1 score of 0.990 for claim extraction, localizes evidence within full texts at a 68% hit rate (98% in abstracts), and delivers an end-to-end claim verification accuracy of 71%.
Nearly three-quarters of citation claims in scientific manuscripts can now be verified automatically directly from source PDFs, transforming one of peer review's most labor-intensive failure modes into an automated pipeline.
As software engineering research submission counts surge, peer reviewers face severe time constraints, making systematic verification of citation-supported claims prohibitively expensive. Consequently, unsubstantiated claims and semantic drift can propagate undetected across scientific literature. Existing approaches such as fact-checking and retrieval-augmented generation tools operate on open-domain web data or evaluate claims in isolation without processing complete manuscripts. To address this gap, we present RefVerifier, a semi-automated, citation-bounded reference verification prototype designed to support in academic peer review. RefVerifier extracts citation-bearing sentences from manuscripts, checks bibliography metadata against scholarly databases, resolves references to full-text open-access PDFs, localizes relevant evidence passages, and generates verdicts with natural language explanations. Evaluating RefVerifier on public benchmarks shows claim detection at an F1 score of 0.990, open-access resolution of 57.6% of references, and evidence localization with a hit rate of 98% on abstracts and 68% on complete cited papers. In an end-to-end test with eight manuscripts, RefVerifier achieves a verdict accuracy of 71%. By automating document retrieval and evidence localization while preserving reviewer oversight, RefVerifier provides first indicators for the feasibility of semi-automated integrity checks in scholarly publishing.