Search papers, labs, and topics across Lattice.
This paper introduces SCURank, a novel framework for ranking candidate summaries generated by multiple LLMs based on the richness and semantic importance of their Summary Content Units (SCUs). SCURank addresses the instability of existing LLM-based ranking strategies and the limitations of surface-level metrics like ROUGE. Experiments show that SCURank outperforms traditional metrics and LLM-based ranking methods, leading to improved abstractiveness and overall performance in distilled summarization models.
Ditch ROUGE and unstable LLM rankings: SCURank leverages Summary Content Units to identify and select the most semantically rich summaries from diverse LLMs, boosting distillation performance.
Small language models (SLMs), such as BART, can achieve summarization performance comparable to large language models (LLMs) via distillation. However, existing LLM-based ranking strategies for summary candidates suffer from instability, while classical metrics (e.g., ROUGE) are insufficient to rank high-quality summaries. To address these issues, we introduce \textbf{SCURank}, a framework that enhances summarization by leveraging \textbf{Summary Content Units (SCUs)}. Instead of relying on unstable comparisons or surface-level overlap, SCURank evaluates summaries based on the richness and semantic importance of information content. We investigate the effectiveness of SCURank in distilling summaries from multiple diverse LLMs. Experimental results demonstrate that SCURank outperforms traditional metrics and LLM-based ranking methods across evaluation measures and datasets. Furthermore, our findings show that incorporating diverse LLM summaries enhances model abstractiveness and overall distilled model performance, validating the benefits of information-centric ranking in multi-LLM distillation. The code for SCURank is available at https://github.com/IKMLab/SCURank.
