Search papers, labs, and topics across Lattice.
This paper introduces VICBench, a comprehensive benchmark dataset designed for evaluating code vulnerability detection tools, consisting of 100 verified vulnerability-inducing commits (VICs) across 88 projects in Python, Java, and C++. The dataset addresses the limitations of existing benchmarks by providing a diverse range of programming languages, complex patch scenarios, and a wide scope of Common Weakness Enumeration (CWE) types. Evaluation of state-of-the-art algorithms reveals that they still achieve low F1 scores (33.3%-40.1%), highlighting the necessity for improved methods in vulnerability detection.
Existing vulnerability detection tools struggle with a mere 33.3%-40.1% F1 score, underscoring the urgent need for better benchmarks like VICBench.
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.
Amazon Science