Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper meta-evaluates 14 automatic summarization metrics and LLM-based evaluators on 7 datasets, finding that LLM-based evaluators outperform traditional metrics in aligning with human judgments, especially for linguistic quality. Based on this, they introduce LLM-ReSum, a self-reflective summarization framework that uses LLM-based evaluation to refine summaries in a closed loop, without finetuning. LLM-ReSum achieves up to 33% improvement in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries 89% of the time.
Forget ROUGE scores: LLM-ReSum shows how to leverage LLMs to iteratively improve summary quality by up to 39% using a closed-loop, self-evaluation framework.
Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and datasets will be released in GitHub.