Search papers, labs, and topics across Lattice.
The authors introduce VibeSearchBench, a new benchmark designed to evaluate LLM-based agents in long-horizon, proactive search scenarios that more closely resemble real-world user interactions. Unlike existing benchmarks, VibeSearchBench features vague, multi-turn interactions with a user simulator and schema-free knowledge graphs, capturing the collaborative refinement of user intent. Experiments with seven state-of-the-art models reveal significant shortcomings in their ability to perform VibeSearch, achieving a best F1 score of only 30.30, indicating a need for advancements in long-context reasoning and intent elicitation.
LLMs still fail at realistic search, with even the best models achieving only 30% F1 on a new benchmark designed to mimic the messy, multi-turn, and vaguely-defined search tasks that real users face.
LLM-based agents score well on search benchmarks, yet real users consistently find results unsatisfying, revealing a persistent evaluation-experience gap. We attribute this gap to existing benchmarks'reliance on over-specified queries, single-turn interactions, and fixed-schema evaluation, none of which reflect real search behavior where users and agents collaboratively refine vague intent through multi-turn dialogue. We term this paradigm VibeSearch and introduce VibeSearchBench, a benchmark comprising 200 manually curated bilingual (Chinese and English) tasks across 20 domains, split into VibeSearch-Pro (professional) and VibeSearch-Daily (daily-life) subsets. Each task pairs a user persona with a schema-free ground-truth knowledge graph, and is evaluated through a progressive-disclosure user simulator and a graph-matching evaluation framework. We benchmark seven frontier models under both the ReAct framework and the OpenClaw agent harness. Results show that all models remain substantially inadequate for VibeSearch (best F1: 30.30), highlighting the need for fundamental advances in long-context reasoning, proactive intent elicitation, and structured knowledge construction.
