Search papers, labs, and topics across Lattice.
This paper introduces StealthBench, a benchmark designed to evaluate the operational stealth of autonomous offensive-security agents across six dimensions of operational security (OPSEC). By analyzing 11 verified OPSEC incidents and expanding them into 14 dockerized scenarios, the authors found that agents frequently exhibited stealth failures, such as exposing credentials and demonstrating access inappropriately. The results revealed that no model achieved a safe success rate exceeding 54%, highlighting systematic OPSEC failures across various model families and underscoring the need for improved stealth in autonomous systems.
Autonomous agents are failing to maintain operational stealth, with no model achieving more than 54% in safe success rates during offensive-security tasks.
Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisticated operators from detectable ones. Elite security researchers and advanced persistent threats achieve their objectives unnoticed; autonomous agents increasingly inherit the same offensive tasks, but do they inherit the tradecraft? We introduce StealthBench,a benchmark that measures operational stealth in autonomous offensive-security agents across six operational security (OPSEC) dimensions. We extract 11 hand-verified OPSEC incidents from real bug-bounty and red-team trajectories, expanded into 14 dockerized task scenarios, where agents, despite finding real vulnerabilities, committed stealth failures inconsistent with standard operational tradecraft: embedding credentials in public uploads, deleting production resources to prove access, force-adding uninvolved users to demonstrate a race condition. We evaluate agent trajectories using a 3-model large language model (LLM) judge panel with majority-vote aggregation, measuring safe success rate (solved and stealthy), Stealth@Solve (tradecraft quality among successful solves), and reckless solve rate (solved but cover blown). Our results show that no model exceeds 54% safe success rate (the compound metric requiring both task completion and stealth), confirming that OPSEC failures are systematic across model families. We release StealthBench as a public benchmark to support both the development of stealth-aware agents and automated OPSEC monitoring for autonomous offensive-security deployments. The interactive leaderboard, evaluation harness, and dataset are available at https://stealthbench.com.
