Search papers, labs, and topics across Lattice.
This paper introduces DailyReport, a novel open-ended benchmark designed to evaluate search agents (SAs) on daily search tasks, addressing the limitations of existing benchmarks that focus on specialized tasks. By incorporating 150 diverse tasks and 3,546 detailed rubrics, the authors enable a more nuanced assessment of SA performance through cascade rubrics and user-centric aggregation methods. Results from 17 different agentic systems reveal that current SAs significantly underperform relative to user expectations, highlighting the need for improved capabilities in real-world scenarios.
Current search agents fall short of user expectations, with a new benchmark revealing critical gaps in their performance on everyday tasks.
Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses. For SAs evaluation, prior benchmarks mainly focus on specialized tasks that are unlikely to arise in real-world user scenarios. Moreover, their reliance on coarse task-level rubrics often limits evaluation interpretability. To bridge this gap, we introduce DailyReport, an open-ended benchmark to evaluate SA capabilities on daily search tasks. It contains 150 open-ended tasks with 3,546 associated rubrics, capturing widely discussed and timely information demands of real-world users. Each task is decomposed into subtasks and evaluated with cascade rubrics across disentangled dimensions. Through cascade performance attribution and user-centric aggregation, we derive highly interpretable scores for each dimension, along with a user preference score. Our results on 17 agentic systems show that current systems still fall short of users' expectations. To facilitate future research, our dataset and code are made publicly available at https://github.com/AGI-Eval-Official/DailyReport.
