Search papers, labs, and topics across Lattice.
This paper introduces EvoHunt, an innovative environment for evolving audit playbooks that enhances vulnerability discovery in code analysis by leveraging a combination of LLMs, agent harnesses, and dynamic procedural knowledge. By automating playbook evolution, the authors demonstrate a significant improvement in vulnerability detection rates, with Codex/GPT5.4-xhigh achieving a 6x increase in end-to-end exploits and the evolved OpenCode/GLM5.1 playbook outperforming OpenAI Codex Security across all metrics. The findings reveal that evolved playbooks can effectively transfer their capabilities to weaker agents, enhancing their performance in security auditing tasks.
Evolved playbooks can boost vulnerability detection rates by over 6x and outperform dedicated commercial products, reshaping the landscape of automated security auditing.
An LLM agent for vulnerability discovery and validation is more than a model. It combines three components: an LLM for code analysis, an agent harness such as Codex or OpenCode for navigation, tool use, and execution, and an audit playbook, domain-specific procedural knowledge that guides the LLM and harness toward vulnerability discovery. Prior work relies on human-supplied playbooks, including prompt engineering, manual workflows, knowledge bases, and heuristics. This raises two research questions: Acquisition - is human curation necessary, and can playbook creation be automated? Transfer - can an evolved playbook transfer the audit procedure to weaker agents, improving their capability? We present EvoHunt, a playbook evolution environment over open-source repositories for security auditing. Three agents drive the evolution loop: an audit agent rolls out the current playbook and produces findings; an evaluator scores outcomes against ground truth; and a reviser commits updates to the playbook based on failure analysis. The playbook format is unconstrained: starting empty, EvoHunt adds or removes workflows, heuristics, vulnerability knowledge, or domain-specific content. The evolved playbook requires only minor adaptation to run under a different LLM or harness. We evaluate EvoHunt on open-source security advisories. For acquisition, playbook evolution raises end-to-end exploits for Codex/GPT5.4-xhigh 6x, from 1.1% to 6.2%, and the evolved OpenCode/GLM5.1 playbook surpasses OpenAI Codex Security on every metric, with 11.3% vs. 9.2% target-match rate, showing open-source evolution can outperform a dedicated commercial product. For transfer, the GLM-evolved playbook gives the strongest student lift: Qwen3.6-27B improves from 2.4% to 6.5%, Qwen3.6-35B-A3B from 1.1% to 4.6%, and A3B obtains 2.4x more matches than GPT transfer.
BAIR