Search papers, labs, and topics across Lattice.
This paper introduces WebRetriever, a comprehensive benchmark designed to evaluate web agents across 800 websites and 1,550 tasks, addressing critical gaps in existing evaluation frameworks. The authors propose NavEval, a novel LLM-as-Judge methodology that captures nuanced interaction semantics and aligns closely with human judgment, significantly enhancing evaluation accuracy. Experimental results reveal that traditional navigation success metrics fail to predict real-world effectiveness, underscoring the need for a multifaceted assessment approach in web agent development.
Navigation success alone is a poor predictor of real-world web agent effectiveness, as revealed by the extensive evaluation of WebRetriever's diverse protocols.
As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.
