Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a benchmark of 20 interactive security targets to evaluate the performance of different agentic architectures using LLMs for offensive security tasks. They systematically compare five architecture families across web/API and binary targets in whitebox and blackbox settings, measuring detection rates and costs. The key finding is that increasing agent coordination does not always improve performance when considering latency, token cost, and exploit validation difficulty, resulting in a non-monotonic cost-quality frontier.
More agents don't always mean better security: LLM-powered offensive security systems hit diminishing returns as agent coordination increases, with simpler architectures often proving more cost-effective.
Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear when additional agents help and when they only add cost. We treat topology choice as an empirical systems question. We introduce a controlled benchmark of 20 interactive targets (10 web/API and 10 binary), each exposing one endpoint-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes. The core study executes 600 runs over five architecture families, three model families, and both access modes, with a separate 60-run long-context pilot reported only in the appendix. On the completed core benchmark, detection-any reaches 58.0% and validated detection reaches 49.8%. MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding. Whitebox materially outperforms blackbox (67.0% vs. 32.7% validated detection), and web materially outperforms binary (74.3% vs. 25.3%). Bootstrap confidence intervals and paired target-level deltas show that the dominant effects are observability and domain, while some leading whitebox topologies remain statistically close. The main result is a non-monotonic cost-quality frontier: broader coordination can improve coverage, but it does not dominate once latency, token cost, and exploit-validation difficulty are taken into account.