Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the trustworthiness of zero-shot summarization produced by Large Language Models (LLMs) by introducing a two-level diagnostic protocol that benchmarks the stability of generated summaries. Through document-level stability analysis and the computation of a stability coefficient, the authors assess semantic and factual alignment across multiple summaries, revealing significant variability in performance among different LLMs. The findings highlight the critical need for improved reliability in LLM-generated summaries, especially in educational contexts where accuracy is paramount.
LLM-generated summaries can vary significantly in trustworthiness, with some models exhibiting up to 40% instability in their outputs across different document types.
Zero-shot summarization using Large Language Models (LLMs) has significantly advanced the abstractive summarization task by producing coherent and fluent summaries. However, underlying stochasticity of the large language models raises concerns about the stability and trustworthiness of the LLM-generated summaries. This issue has become increasingly important due to proliferation of LLM-generated summaries in educational settings, where students and researchers summarize complex academic materials in zero-shot manner. We propose a novel two-level diagnostic protocol for benchmarking LLM-summarizers based on the stability of the generated summaries. At the lower level, document-level stability analysis is performed over multiple LLM-summaries generated under controlled environment, and the stability coefficient is computed. Each generated summary is scored for semantic and factual alignment with the original document, enabling estimation of stability along more than one dimensions. At the next level, observations from a stratified sample of documents drawn from the corpus are consolidated to estimate the stability index of the LLM-summarizer, which is the proxy for its trustworthiness. Our empirical investigation of three LLM-summarizers across three genres of documents reveals statistically significant differences in the generation-level variability among LLMs across summary evaluation metrics. This study advances the LLM-summarization research by evidential recognition of the stability problem in LLM-summaries and motivates further research towards development of robust, reliable and trustworthy LLM-summarizers.