Search papers, labs, and topics across Lattice.
This paper introduces a two-stage decoupled evaluation framework for assessing the capabilities of LLM-based penetration testing agents, effectively separating the processes of reconnaissance and exploitation. By evaluating five different architectures across 70 web vulnerability testbeds, the authors reveal a significant performance gap: agents achieve up to 90% success in exploiting known vulnerabilities when provided with accurate context, while autonomous reconnaissance only reaches about 50% recall due to challenges in processing unstructured data. The findings highlight the importance of precise reconnaissance in maximizing exploitation success and suggest architectural optimizations for different types of vulnerabilities.
LLM-based penetration testing agents can achieve up to 90% success in exploiting vulnerabilities, but their reconnaissance capabilities plateau at 50%, revealing a critical gap in automated security assessments.
Large Language Models (LLMs) have shown promise for automated penetration testing, yet existing end-to-end black-box evaluations are highly susceptible to error cascading: failures in early reconnaissance can mask an agent's actual ability to exploit vulnerabilities. To more accurately characterize these capabilities, we propose a two-stage decoupled evaluation framework that separates exploit execution from reconnaissance. Using ground-truth injection and knowledge-driven ablation across 70 high-fidelity web vulnerability testbeds, our framework isolates exploitation performance from reconnaissance noise. We empirically evaluate five open-source penetration-testing agents, covering multiagent, monolithic, and graph-driven architectures, on a strictly aligned subset of 50 representative vulnerabilities. The results reveal a substantial capability gap. With accurate vulnerability context, agents achieve a functional success rate of up to 90.0%, whereas autonomous reconnaissance, measured by targeted vulnerability recall, plateaus at approximately 50.0%, primarily due to failures in parsing unstructured telemetry. Cross-architectural analysis further reveals distinct capability niches: multi-agent isolation is more effective for long-sequence interactions such as de-serialization, while monolithic and graph-driven designs perform better on short-chain injections and cross-session access-control vulnerabilities, respectively. This decoupled evaluation work provides a fine-grained benchmarking protocol and an empirical basis for designing next-generation automated offensive security agents.
