Search papers, labs, and topics across Lattice.
This paper introduces a novel evaluation protocol for AI pentesting agents that transitions from traditional task completion metrics to a focus on validated vulnerability discovery in complex real-world scenarios. By integrating structured ground-truth data with LLM-based semantic matching and bipartite resolution, the authors create a robust framework that accommodates the inherent ambiguity of real-world pentesting. The key finding reveals that this new protocol significantly enhances the operational relevance of performance comparisons among AI pentesting agents, paving the way for more effective security assessments.
Shifting the focus from task completion to real-world vulnerability discovery could revolutionize how we evaluate AI pentesting agents.
AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets. Existing evaluation protocols assess and optimize for predefined goals such as capture-the-flag, remote code execution, exploit reproduction, or trajectory similarity, in simplified or narrow settings. These tools are valuable for measuring bounded capabilities, yet they do not adequately capture the complexity, open-ended exploration, and strategic decision-making required in realistic pentesting. In this paper, we present a practical evaluation protocol that shifts assessment from task completion to validated vulnerability discovery, allowing evaluation in sufficiently complex targets spanning multiple attack surfaces and vulnerability classes. The protocol combines structured ground-truth with LLM-based semantic matching to identify vulnerabilities, bipartite resolution to score findings under realistic ambiguity, continuous ground-truth maintenance, repeated and cumulative evaluation of stochastic agents, efficiency metrics, and reduced-suite selection for sustainable experimentation. This protocol extends the state of the art by enabling a more realistic, operationally informative comparison of AI pentesting agents. To enable reproducibility, we also release expert-annotated ground truth and code for the proposed evaluation protocol: https://github.com/ethiack/ethibench.
