Search papers, labs, and topics across Lattice.
This paper introduces the Open Security Benchmark (OSB), a framework designed to evaluate agentic AI in autonomous cyber defense by providing a curated, immutable snapshot of an enterprise's security state. By addressing the "environment data gap," OSB allows for comprehensive assessments of AI agents' trustworthiness in making security decisions across diverse modalities, including text-to-SQL and vendor APIs. The framework is instantiated with identity-security packs and synthetic datasets, paving the way for future extensions into various security subdomains and evaluation methods.
Trusting AI in cyber defense hinges on a benchmark that captures the real complexity of enterprise environments鈥擮SB fills this critical gap.
Enterprises are moving toward autonomous cyber defense: agentic AI that builds situational awareness of an organization's security state and reasons from it to assessments, decisions, and actions. This rests on a holistic view of the enterprise's security state, the continuous, cross-vendor picture of identities, cloud and infrastructure, data, applications, and their configurations that security posture management assembles. As agents take on this work, what matters is not whether an agent can produce an answer but whether it should be trusted to. The field cannot yet answer this question. Real enterprise environments are private, cross-vendor, and deeply correlated, and none is exposed publicly as a shared, queryable target for evaluating such agents end to end. We call this the environment data gap. We present Open Security Benchmark (OSB), a framework that benchmarks agentic AI on this work. OSB surfaces a curated enterprise environment - a frozen, holistic view of the security state - and evaluates posture investigation across two modalities: text-to-SQL over a relational snapshot and each vendor's native API over a served instance of the same environment. Freezing the environment pins the target state as an immutable snapshot and anchors answers to a closed-form ground truth. OSB is built from five components: a data layer, a task and evaluation-set layer, a multi-dimensional scoring layer, a minimal auditable harness, and a bring-your-own path that serves public comparison and private tenant evaluation from one substrate. We instantiate the framework with two identity-security packs and a family of synthetic-organization environment datasets spanning multiple scales, and chart its extension to further posture subdomains, investigation modalities, and defense stages from assessment toward remediation.
