Search papers, labs, and topics across Lattice.
This paper introduces a tri-agent framework designed to evaluate and enhance the question clarification capabilities of large language models (LLMs) in interactive systems. The framework consists of three agents: a Question Clarifying Agent that identifies ambiguities, a Respondent Agent that simulates user responses, and an Evaluator Agent that assesses dialogue quality using various metrics. Key findings indicate that this structured approach not only benchmarks LLM performance effectively but also provides insights for improving conversational AI applications in understanding user intent.
LLMs can significantly enhance user interactions by mastering question clarification, as revealed by a novel tri-agent evaluation framework that benchmarks their performance.
Large Language Models (LLMs) are increasingly deployed in interactive systems where understanding user intent precisely is paramount. A key capability for such systems is effective question clarification, especially when user queries are ambiguous or underspecified. This paper introduces a novel tri-agent framework for the robust evaluation of an LLM's ability to engage in clarifying dialogue. Our framework comprises three distinct LLM-based agents: (1) a Question Clarifying Agent (QCA), the system under evaluation, tasked with identifying ambiguities and posing clarifying questions; (2) a Respondent Agent (RA), designed to simulate human user responses, potentially including irrelevant or challenging replies; and (3) an Evaluator Agent (EA), an LLM-as-a-judge, which assesses the quality of the dialogue based on a comprehensive set of metrics. We detail a methodology for synthetic data generation in the supply chain domain as an example. We propose metrics evaluating ambiguity handling, question quality, dialogue efficiency, language appropriateness, and final intent alignment. We also briefly discuss the validation of the EA against human judgments. This work provides a structured approach to benchmark, validate, and improve the clarification capabilities of conversational LLM applications.
