Search papers, labs, and topics across Lattice.
The paper introduces WebSP-Eval, a new benchmark for evaluating web agents on website security and privacy tasks, addressing a gap in existing benchmarks that focus on general performance or malicious actions. They created a dataset of 200 task instances across 28 websites, a robust agentic system, and an automated evaluator. Experiments with 8 web agent instantiations revealed limitations in autonomous exploration and difficulties with stateful UI elements like toggles and checkboxes, which caused failures in over 45% of tasks containing them.
Web agents are surprisingly bad at basic website security and privacy tasks, failing almost half the time when confronted with common UI elements like toggles and checkboxes.
Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries. While current benchmarks evaluate general-purpose performance~(e.g., WebArena) or safety against malicious actions~(e.g., SafeArena), no existing framework assesses an agent's ability to successfully execute user-facing website security and privacy tasks, such as managing cookie preferences, configuring privacy-sensitive account settings, or revoking inactive sessions. To address this gap, we introduce WebSP-Eval, an evaluation framework for measuring web agent performance on website security and privacy tasks. WebSP-Eval comprises 1) a manually crafted task dataset of 200 task instances across 28 websites; 2) a robust agentic system supporting account and initial state management across runs using a custom Google Chrome extension; and 3) an automated evaluator. We evaluate a total of 8 web agent instantiations using state-of-the-art multimodal large language models, conducting a fine-grained analysis across websites, task categories, and UI elements. Our evaluation reveals that current models suffer from limited autonomous exploration capabilities to reliably solve website security and privacy tasks, and struggle with specific task categories and websites. Crucially, we identify stateful UI elements such as toggles and checkboxes are a primary reason for agent failure, failing at a rate of more than 45\% in tasks containing these elements across many models.
