Search papers, labs, and topics across Lattice.
This paper introduces a meta-scaffold, Cybersecurity SuperIntelligence (CSI), to unify and benchmark different LLM-driven agent harnesses for cybersecurity tasks. They benchmarked five scaffolds on 33 cybersecurity challenges, finding that no single scaffold dominates and that a combination of heterogeneous scaffolds yields the best performance. CSI's blackboard-based multi-agent architecture, which allows specialized agents to run in parallel and exchange findings, achieves a 27% relative gain over the best individual scaffold and is 25% faster.
Forget monolithic cybersecurity AI: a diverse team of specialized LLM agents, sharing insights on a digital blackboard, crushes solo performance.
What is the best harness for cybersecurity AI? Cybersecurity systems are converging on a single execution scaffold per agent, an iterative shell loop driven by a Large Language Model (LLM). However, scaffolds are not interchangeable, rarely interoperable, and no single scaffold dominates across all challenge types. In our path towards researching Cybersecurity SuperIntelligence (CSI), we present a meta-scaffold that unifies heterogeneous agent harnesses under a common orchestration layer, enabling any LLM-driven scaffold to be deployed, benchmarked, and composed within the same infrastructure. Using CSI, we benchmark five scaffolds (CSI::Claude, CSI::Codex, CSI::GCAI, CSI::Mistral, CSI::CAI) on the 33 cybench challenges, holding the model fixed at alias2-mini. The best individual scaffolds solve 15/33 (45.5%); the four-scaffold union solves 17/33 (51.5%), with the fifth (CSI::Mistral, 10/33) contributing one exclusive solve. We find that no single scaffold is the best harness: it is the combination of structurally heterogeneous scaffolds that yields the highest coverage. We validate this through CSI's blackboard-based multi-agent architecture, in which scaffold-specialised agents run in parallel and exchange intermediate findings via a shared substrate (a blackboard). The blackboard solves 19/33 (57.6%), a 27% relative gain over CSI::Claude, one of the best individual scaffolds (15/33, 45.5%), 25% faster (20.2 h vs. 26.8 h), at comparable cost ($5,480 vs. $5,122).
