Search papers, labs, and topics across Lattice.
This paper introduces ATOBench, an evaluation framework designed to make the verification process of autonomous penetration-testing agents observable by injecting registered response transformations at runtime. The framework allows for a detailed analysis of how agents interpret conflicting evidence and adjust their actions, revealing that increased activity can obscure verification failures while successful evidence recovery is critical for accurate reporting. The evaluation of five model routes over 450 episodes demonstrates that ATOBench effectively highlights the impact of deceptive target responses on the agents' decision-making processes and reporting accuracy.
ATOBench reveals that deceptive responses can obscure verification failures, fundamentally altering how autonomous penetration-testing agents interpret evidence and report vulnerabilities.
Autonomous penetration-testing agents rely on target responses. These responses guide both subsequent actions and the final report. A deceptive response can therefore redirect both the attack trajectory and the agent's verification process. However, final reports reveal little about how an agent interprets conflicting evidence, changes course, decides to stop, or turns observations into a vulnerability claim. We introduce ATOBench, an evaluation framework that makes this verification process observable. ATOBench injects registered response transformations at runtime and pairs each transformed episode with a native episode under the same environment. Each pair is aligned at the first affected response. A source-linked reconstruction then follows later actions, evidence recovery, stopping, and report support. Three frozen observation contracts cover exploit proof, resource ownership, and reusable artifacts. We evaluate five model routes over 450 episodes. The analysis shows that increased activity can mask a broken verification chain, while successful recovery depends on finding usable evidence and preserving it through reporting. ATOBench turns deceptive target observations into a reproducible probe of evidence handling in autonomous penetration testing. This process-level view extends offensive pentest agent evaluation beyond final outcomes by revealing how untrusted observations shape actions, verification, and reporting.
