Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel set of metrics—Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR)—to quantitatively assess the abstractiveness of text summarization models, moving beyond traditional ROUGE metrics. By employing a harmonic mean modulated by a cubic non-overlap factor, these metrics provide dimensionally consistent outputs that effectively differentiate between extractive and abstractive summarization approaches. Empirical validation on 100 XSUM documents reveals that while extractive models yield low SA scores (0.12-0.26), abstractive models achieve significantly higher scores (0.96-1.77), highlighting the metrics' ability to flag summaries that may require further evaluation for hallucination risks.
New metrics reveal stark differences in abstractiveness between summarization models, with potential implications for identifying hallucinations in generated summaries.
Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.