Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a framework for evaluating structured summaries generated by large language models and placed above organic web search results. The framework assesses the quality of these summaries based on overview quality, section relevance, citation accuracy, and source document utility. The authors outline their plans for implementing and evaluating this framework, aiming to improve the user experience of web search.
LLM-generated search summaries could be rigorously evaluated for overview quality, section relevance, and citation accuracy using the framework proposed here.
We propose a framework for evaluating structured generative search summaries that are placed atop organic web search results. A structured summary, generated by a large language model, typically consists of an overview, several sections with section titles, and a list of source documents that are cited within the summary. We then describe our plans for implementing and evaluating the framework.