Search papers, labs, and topics across Lattice.
The paper introduces NovBench, a new benchmark for evaluating LLMs on academic paper novelty assessment, consisting of 1,684 paper-review pairs from an NLP conference. They propose a four-dimensional evaluation framework (Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations, comparing LLM outputs against novelty descriptions from paper introductions and expert-written evaluations. Experiments with general and specialized LLMs reveal that current models struggle with scientific novelty comprehension and instruction following, highlighting the need for improved fine-tuning strategies.
LLMs still can't reliably judge the novelty of research papers, even when fine-tuned on peer review data, suggesting current models lack a deep understanding of scientific contributions.
Novelty is a core requirement in academic publishing and a central focus of peer review, yet the growing volume of submissions has placed increasing pressure on human reviewers. While large language models (LLMs), including those fine-tuned on peer review data, have shown promise in generating review comments, the absence of a dedicated benchmark has limited systematic evaluation of their ability to assess research novelty. To address this gap, we introduce NovBench, the first large-scale benchmark designed to evaluate LLMs'capability to generate novelty evaluations in support of human peer review. NovBench comprises 1,684 paper-review pairs from a leading NLP conference, including novelty descriptions extracted from paper introductions and corresponding expert-written novelty evaluations. We focus on both sources because the introduction provides a standardized and explicit articulation of novelty claims, while expert-written novelty evaluations constitute one of the current gold standards of human judgment. Furthermore, we propose a four-dimensional evaluation framework (including Relevance, Correctness, Coverage, and Clarity) to assess the quality of LLM-generated novelty evaluations. Extensive experiments on both general and specialized LLMs under different prompting strategies reveal that current models exhibit limited understanding of scientific novelty, and that fine--tuned models often suffer from instruction-following deficiencies. These findings underscore the need for targeted fine-tuning strategies that jointly improve novelty comprehension and instruction adherence.
