Search papers, labs, and topics across Lattice.
This paper introduces RegretBench, a novel multi-turn benchmark designed to evaluate the clarification policies of conversational LLMs by framing ambiguity as a sequential decision-making problem. By employing a regret-based objective, the benchmark assesses the value lost when models deviate from an optimal clarification strategy, highlighting the importance of not just accuracy but also efficiency and robustness in user interactions. Experiments demonstrate that models can achieve similar accuracy while exhibiting significant differences in their clarification effectiveness, revealing that timely and contextually appropriate questioning is crucial for successful user intent resolution.
Models that seem equally accurate can drastically differ in their ability to clarify ambiguous requests, impacting user experience and efficiency.
Ambiguous user requests make clarification a sequential decision problem for conversational LLM assistants: they must decide whether to ask, what to ask, when to stop, and when to answer. We introduce RegretBench, a multi-turn benchmark that evaluates clarification as policy behavior rather than isolated question quality. RegretBench provides a hidden-intent formulation of ambiguity, supports free-form interaction grounded in semantic-state tracking, and introduces a regret-based objective that measures how much value a model loses relative to a reference clarification policy. Experiments on open-domain QA and product recommendation scenarios show that final success alone is insufficient, as models with similar accuracy can differ substantially in efficiency, robustness to user behaviors, and stopping decisions. By jointly measuring intent resolution, interaction cost, ineffective clarification, and regret, RegretBench reveals whether models clarify usefully and efficiently. Our results show that effective clarification requires more than plausible questions: models must ask the right question at the right time and stop once the user's intended meaning is clear.
