Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study systematically evaluates the effectiveness of LLM-generated rubrics for assessing paper reproduction, addressing the scalability challenges of expert-created benchmarks like PaperBench. By reformulating rubrics into a checklist format and testing four generation settings across two backbone models, the authors found that the best-performing LLM-generated rubrics achieved score alignment with human-created rubrics that approached the baseline. However, the analysis also revealed that these rubrics tend to be overly detailed, biased towards higher scores, and less adaptable to different paper domains, indicating both their potential and limitations in practical applications.
LLM-generated rubrics can nearly match human evaluation standards, but they often miss the mark with excessive detail and scoring bias.
Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucination. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the affordances and limitations.