Search papers, labs, and topics across Lattice.
This paper introduces DiscoBench, a benchmark specifically designed to evaluate clarification-aware deep search agents in the context of ambiguous user queries. The research highlights that existing benchmarks fail to account for the inherent vagueness and factual inaccuracies in real-world search requests, which can lead to incorrect reasoning paths. Experimental results reveal that effective ambiguity detection and clarification questioning are crucial capabilities that current large language models (LLMs) often lack, with direct clarification outperforming repeated searching in many scenarios.
Ambiguity detection and effective clarification questioning are often neglected capabilities in LLM-powered search agents, leading to significant performance gaps.
Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.
