Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces RealVuln, a benchmark for evaluating security scanners on real-world Python code containing intentionally planted vulnerabilities, comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized tools. Fifteen scanners were tested across 26 repositories with 796 hand-labeled entries, revealing a three-tier performance ranking based on F3 score, with Security-Specialized scanners leading, followed by General-Purpose LLMs, and then Rule-Based tools. The benchmark, including code, data, and scoring scripts, is open-sourced and designed to be a living, community-driven resource.
Security-specialized scanners beat general-purpose LLMs by a wide margin when hunting for real-world code vulnerabilities, but LLMs still crush traditional rule-based tools.
How do security scanners perform on real-world code? We present RealVuln, the first open-source benchmark comparing Rule-Based SAST, General-Purpose LLMs, and Security-Specialized scanners on 26 intentionally vulnerable Python repositories (educational and Capture-The-Flag applications) with 796 hand-labeled entries (676 vulnerabilities, 120 false-positive traps). We test 15 scanners (3 Rule-Based SAST, 10 General-Purpose LLM, 2 Security-Specialized) and rank them by F3 score (beta=3, weighting recall 9x over precision). A clear three-tier ranking emerges under all metrics. Under F3, the Security-Specialized scanner Kolega.Dev (73.0) leads, followed by the best General-Purpose LLM, Claude Sonnet 4.6 (51.7), which in turn scores nearly 3x higher than the best Rule-Based tool, Semgrep (17.7). Under F1, Sonnet 4.6 leads (60.9) with Kolega.Dev at 52.4. Rankings within tiers shift with beta, but the three-tier hierarchy holds across all weightings. All code, ground-truth data, scanner outputs, and scoring scripts are released under an open-source license. An interactive dashboard is at https://realvuln.kolega.dev/. RealVuln is a living benchmark: versioned, community-driven, with a roadmap toward multi-language coverage.