Search papers, labs, and topics across Lattice.
This paper introduces AutoSupervision, a framework designed to verify the effectiveness of manuscript revisions in addressing reviewer feedback by utilizing peer-review records as a source of supervision. The study reveals that while large language models (LLMs) excel at identifying reviewer concerns, they struggle significantly with the evidence-based verification of whether those concerns have been adequately addressed. Notably, the best-performing model achieved a verification score of only 0.501, highlighting a critical gap in the capability of LLMs to support reliable scientific workflows.
Despite LLMs excelling at identifying reviewer concerns, they falter in verifying if revisions truly resolve those issues, with the best achieving only a 0.501 score in evidence-based checks.
Recent advances in large language models (LLMs) have enabled AI systems to assist scientific research and peer review. However, an essential capability for reliable AI-assisted scientific workflows remains underexplored: verifying whether reviewer feedback leads to meaningful and evidence-supported manuscript improvements. We introduce AutoSupervision, which evaluates whether scientific manuscript revisions genuinely address reviewer concerns through grounded evidence. AutoSupervision leverages transparent peer-review records as a natural source of supervision, where reviewer comments specify scientific concerns, author responses describe claimed resolutions, and revised manuscripts provide evidence of changes. Given reviewer comments, author responses, and revised manuscripts, models must characterize reviewer concerns, determine whether concerns have been addressed, and identify supporting manuscript evidence. We construct AutoSupervision from 56,000 Nature Communications articles and corresponding review records. Then we conducted experiments on LLMs, the ablation study, and the case study. Our results show that while LLMs perform well in characterizing reviewer concerns, with GPT-5.5 achieving a score of 0.754, evidence-based verification remains the primary bottleneck, with the best-performing model reaching only 0.501.
