Search papers, labs, and topics across Lattice.
This paper introduces Code-driven Agentic Testing (CAT), a novel framework that enables agents to autonomously write Playwright code for end-to-end GUI testing of web applications. By integrating Browser-Use and Computer-Use tools, CATJudge allows for comprehensive bug discovery in AI-generated web applications, which are benchmarked through CATTest鈥攁 dataset of 102 applications with annotated bugs. Experimental results reveal that mainstream vision-language models (VLMs) significantly underperform in this testing paradigm, highlighting a critical gap in their capabilities for real-world web development scenarios.
Vision-language models struggle to uncover bugs in web applications, revealing a stark gap in their testing capabilities.
End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks, leaving the bug-discovery ability of vision-language models (VLMs) systematically untested. We introduce \textbf{C}ode-driven \textbf{A}gentic \textbf{T}esting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs show that all evaluated models perform poorly, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development. We release our code and data at https://github.com/SleepyWithoutCoffee/CATJudge.
