Search papers, labs, and topics across Lattice.
This study conducts the first systematic meta-evaluation of LLM-generated rubrics for assessing paper reproduction, reformulating traditional rubrics into a checklist-style format. By evaluating four generation settings across two backbone models, the authors find that augmented settings significantly enhance downstream evaluation alignment, with the best-performing configuration nearing human baseline performance. However, the analysis also uncovers that these rubrics tend to be overly detailed, biased towards higher scores, and less adaptable to specific paper domains, revealing both their potential and limitations.
LLM-generated rubrics can nearly match human evaluators in assessing paper reproduction, but they often miss the mark with excessive granularity and bias.
Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repository comparison is prone to hallucination. However, constructing paper-specific rubrics requires substantial expert effort, limiting the scalability of benchmarks such as PaperBench. In this work, we present, to our knowledge, the first systematic meta-evaluation of LLM-generated rubrics for paper reproduction. We reformulate rubrics into a checklist-style format and evaluate four generation settings across two backbone models. We meta-evaluate generated rubrics intrinsically by semantic similarity and extrinsically by score alignment with ground-truth rubrics. Our results show that the augmented settings substantially improves downstream evaluation alignment, with the strongest setting approaching the human baseline, while intrinsic gains are more modest. Further analyses reveal that LLM-generated rubrics are often overly fine-grained, biased toward high scores, and less adaptive to paper domains, highlighting both the affordances and limitations.
