Search papers, labs, and topics across Lattice.
This paper introduces VulnGym, a comprehensive benchmark designed to evaluate coding agents' ability to detect vulnerabilities at the repository level, moving beyond the limitations of isolated code snippet assessments. By aligning GitHub advisories with vulnerable repositories and providing detailed annotations, VulnGym enables a more nuanced evaluation of agents' performance in locating vulnerabilities and constructing supporting evidence. The findings reveal that existing coding agents struggle with both end-to-end vulnerability detection and the generation of accurate traces, highlighting significant gaps in their capabilities.
Current coding agents falter in repository-level vulnerability detection, missing critical evidence construction despite advancements in LLM-based methods.
Recent advances in LLM-based vulnerability detection have shown promising results, while coding agents further extend this capability from isolated code snippets to complete repositories. This shift requires agents to autonomously explore repositories and locate vulnerability-relevant code, instead of performing detection on preselected functions. However, existing benchmarks primarily focus on vulnerability classification over preselected code snippets, limiting their ability to evaluate coding agents in repository-level vulnerability detection. Moreover, without fine-grained vulnerability trace annotations, the capability limitations underlying the detection process remain difficult to explore. To address these limitations, we present \textbf{VulnGym}, a real-world repository-level benchmark for evaluating vulnerability detection by coding agents. VulnGym aligns reviewed GitHub advisories with their corresponding vulnerable version repositories. It contains 184 advisories and 408 vulnerability entries across 23 repositories, with each entry annotated with line-level entry points, critical operations, and vulnerability traces. Using this fine-grained ground truth, VulnGym defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction. Our evaluation indicates that current coding agents remain limited in both end-to-end repository-level vulnerability detection and the construction of accurate supporting traces.
